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This report presents sgi assessment of the 
effectiveness of the first implementation of 
the United States Naval Academy muiti-media 
course in Leadership, Psychology and Manage- 
ment. A description of the course, of the 
development pf materials and tests, of the 
media and presentation forms used, and of 
the procedures for course revision is pro- 
vided. A brief description of the instruc- 
tional research involved in the project is 
also given. 
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.1. INTRODUCTION 

The purpose of the first implementation of the USNA 
Leadership Management Course was to provide empirical data 
upon which the course. coujd be objectively revised. In 
addition to assessing the effectiveness of the course 
materials in teaching stated behavioral objectives, the ) 
effectiveness of various media and presentation forms 
and their relationship to various student characteristics 
was experimentally investigated* 

The evaluation of total course effectiveness was 
based on objective data collected on student performance 
over a series of behavioral objectives. The relative" 
effectiveness of media and pjre^sentation forms was assessed 
from objective data based on a* series of controlled 
comparisons of media and presentation forms within the 
instructional sequence. 

• In this report, only course effectiveness estimated 
from student performance on behavioral objectives will be 
presented. The effectiveness of media and presentation 
forms and their relationship to student characteristics- 
will be presented in the' R eport' Phase' II' Rfesba rth 
- Findings (TR-6.12a § b) . Effectiveness of the final impJe 
mentation system as well as the cost effectiveness of 
the system will be presented in the Phase III Final Report 



Methods used in evaluating course effectiveness 
have been drawn- largely from Technical Recommendations 
from the U. S. Office of Education. For a single ■ 
. volume coverage of evaluation methods actually employed., 
as well as alternate methods, see Some Statistical 
Problems in the Evaluation of Instructional Programs 
(Brennan, 197))". 

Evaluation methods used in the first validation were, 
selected on the basis of their capabilitites for yielding 
meaningful data for course revision. For this^ reason* 
methods such as control comparison of the experimental 
Course vs. an existing course or follow-up study of course 
graduates were not utilized. Since results of these 
studies do not indicate v;here or how to revise course 
materials, these methods are appropriately used only after 
• materials are finalized. I ' • 

A description of the instructional system being 
evaluated is .^'ven in Section II, Media and tests which 
were used are given in Sections III and IV, respectively. 
Evaluation methods which were used are described throughout 
the results sections" of this report. (Sections V, VI, and 
VII). The use of evaluation data in determining the 
adoption-rejection of course components and in identifying 
course components requiring revision is given in 
Section VI Jl. 
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The course has been revised on the basis of the data 
given in this report a id the Report of the Phase II 
Research. Findings. The Leadership, Psychology and 
Management course is being implemented again this fall 
with a replication of the instructional research. The 
results of the second course implementation will be 
.given in the Phase III Final Report. 
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II. DESCRIPTION OP COURSE 

For. the benefit of readers unfamiliar with the USNA 
multi-media Leadership course, a brief description of the 
first implementation^pf the course and the terminology 
used throughout is given in this section. 

COURSE CONll-NT AND OBJECTIVES 

The content outline for the USNA Leadership \ 
Management course was. developed jointly by WLC and the 
USNA Naval Science Department. Sources for the content 
outline were V/LC and USNA subject matter experts, texts 
on managment and naval science, and excerpted materials 
used in the traditional USNA Leadership Management course. 
Reference sources were presented in bibliography form 
at the end of the content outline for each part of the course. 

Having decided the nature of the content, the 
content was analyzed and rough drafts of behavioral 
objectives were developed. In the process of developing 
objectives, the actual sequencing and outlining of total 
course content was completed. Accordingly, the course 
v;as composed of 12 parts (chapters) subdivided into 59 
segments (lessons). A complete listing of part titles 
and segment headings is included in Appendix A (Course 
Outline). The number of segments per part ranged from 
two in Parts Two, Nine and Ten, to ton in Part Five. 



. The sequencing of tbpics within each level of , the 
course was generally determined by SME input and precisely 
determined by behavioral analysis. See Sequencing 
Rationale , Specific behavioral objectives v;ere developed 
at. t\io levels for each topic heading v/ithin a segment. 
Terminal objectives (high level problem solving objectives) 
w^re developed for each major heading. Enabling objectives 
.(lower level objectives) were developed where necessary 
to ensure the learning of information requisite to the 
attainment of terminal objectives. In the actual writing 
of behavioral objectives, V/LC drew heavily from the 
guidelines offered by Robert Mager in Pr ep ar in g Ins t ructi onal 
Objectives . To ensure that behavioral objectives were 
representative of the varying levels or types of learning 
which should occur in the total curriculum, WLC devised 
a classification system to guide the v/riters. The 
classification system represented a composite of systems 
suggested by Bloom's taxonomy (1956), Gagne's eight types 
of learning (1965), and Evans, Homme and Glaser's RULEG 
System (1962). The guidelines are given in Appendix B. 

INSTRUCTIONAL RESEARCH 

Concommitant with the identification and sequencing 
of course content and objectives, WLC developed a series 
of instructional research hypotheses which were incorporated 



in the first implementation of the course. A detailed 
explanation of the instructional research design can be 
found in the Report of Phase II Research Findings, Part I: 
Conditions of Instruction (TR-6.12a). In this report 
only the basic dimensions of the research are given. 

Research hypotheses were based' on concepts presented 
in A Behavioral Approach to Instructional Design and Media 
^Selection, in which a distinction is made between the 
importance of media and presentation forms (Tosti and Ball, 
1968). The distinction is that a medium is only the mode 
of transmission of infoi'mation whereas the truly important 
characteristics of instruction are the forms of presentation 
of the information within the medium. Any medium can be 
described with respect to its characteristic presentation 
dimensions and with respect to its capability for varying 
dimensions of presentation. The most basic dimensions of 
presentation which characterize, a medium are the form and 
frequency of the stimulus, response^ and management decisions. 
In matrix form. these dimensions are: 



Stimulus 
Response 



Management 



DIMENSIONS OF PRESENTATION 



Form 



Stimulus representation 



Response demand form 



Management form 



Frequency 



Duration 



1 



Response demand frequency 
Management frequency 



Stimulus roprosenta'tion is the form in which the 
stimulus is presented. It can be written, spoken, or 
pictorial. 

Stimulus' duration is the temporal stability "of the 
stimulus. Duration can be transient - such as movies and 
lectures, or persistent - such as textbooks or other 
printed matter. 

Response demand form is the type of response the 
subject is required to make.. The form of the response 
demanded can.be covert, written, or spoken. 

Response demand frequency is the frequency with 
which the subject is required to respond. . The frequency 
with which questions are asked within an instructional 
sequence can be high, medium, or low. 

Management form is the type of remiediation the 
subject is given. It can be multilevel - remediation, by 
differing levels of response demand frequency within a 
single form; multiform - remediation by a different form 
of presentation; repetition - a simple repeat of the same 
presentation; or error diagnostic - remediation by 
branching according to specific incorrect responses. 

Management frequency is the frequency with which 
the presentation is repeated or changed according to the 
need for remediation. The frequency of management can be 
high, medium, or low depending on response demand frequency. 



In line with the assumption that presentation 

variables were the critical elements in instruction 

rather than the medium itself, WLC developed five 

hypotheses, four of which were based directly on 

variations in presentation rather than variations in 

media. The hypotheses were: 

Hypothesis 1: Given both simple and 
complex tasks, transient presentations* with 
high response demand frequency (HRDF) will be 
more effective than transient presentations 
with low response demand frequency (LRDF) / - 
There will be no difference in student 
performance between covert or overt response 
demand (RD) presentations, or between 
videotape and audiotape-panelbook media. 

Hypothesis 2: Given both simple and 
complex learning tasks,, persistent presenta- 
tion with high RDF will be more effective 
than persistent presentation with low RDF. 
There will be no difference in student 
performance bet\v'een " three conditions of RD 
(overt-written, overt-spoken, and covert). 

Hypothesis 3: High RDF will be more 
effective than low RDF with either' high, 
medium, or low management frequency; 

Hypothesis 4: If performance is below 
criterion level with low RDF presentation, 
remediation vdth high RDF will result in a 
significant improvement in performance. 

Hypothesis 5: The use of **pecr-monitor" 
procedures will significantly improve student 
performance in an instructional sequence. 

Results of experiments based on these hypothes 

will be presented in the Report of Phase II Research 



Findings (TR-6.12a § b) . 



MHDIA SRLECTION , 

} 

1 

Media selected for the first validation of the 
instructional system were based on three major considera- 
tions i research design requirements, multi -media 
requirements, and implementation requirements. 

1. Media included in the 'instructional research 
design were selected on the basis of their capabilities 
for varying presentation according to requirements of 
the experimental hypotheses. For example, hypothesis 1 
required transient stimulus duration in addition to 
media comparison; therefore, videotapes and audiotapes 
were selected. By using both audiotapes and videotapes 

in parallel modules over the same content, it was possible 
to control all irrelevant presentation variables and 
study the effects of transient presentation across "two 
media as well as comparing the effectiveness of the two 
media. • 

2. In addition to selecting media for their 
research capabilities, media were selected which would 
provide diversification in the instructional sequence. 
Although it was not the intention to directly compare 
media throughout the course, i.e.. Medium A vs. Medium B, 
.or Medium C .vs. Medium D, a number of media were included 
so that some statement could be made about student 
performance via those media. 



3. The third major., consideration in media 
selection was the project plan for implementing and 
maintaining the multi-media system beyond the design 
stage. Of primary concern were the costs for developing, 
implementing, revising and maintaining the system. For 
example, material for certain media could be programed 
during the design stage of the project. but would be 
prohibitively expensive to implement or .maintain (revise) 
after the termination of the contr-act, ]- 

Another concern was that media fit conveniently 
into an individually paced program of studies xcithout 
creating undue logistical problems. In this regard, 
special attention vias given to existing facilities 
at the USNA, such as computer consoles, media centers, 
etc., and to the project availability of those facilities. 

Media included in the first validation were: 
videotaped and audiotaped lectures with panelbooks, 
syndactic text, linear programed text, coi/.puter assisted 
instruction, learning activity summaries (bibliographies), 
film and group discussion. Detailed identification of 
the above media is given in Section III. 

COURSE ORGANIZATION 

The course organization was basically derived from 
a subdivision of content objectives and administration 
of tests based on those subdivisions. Content subdivisions 
are the part and segment. Two course units created as 



a result o£ the instructional research vere the raodule an 

the cumulative posttest unit. 

Content 

A part is equivalent to a chapter in the course ♦ 
It is the largest content division and can be viewed in 
terms of global content objectives. 

A segment is equivalent to a single lesson of 
approximately 40-60 minutes within a part. It is a 
collection of learning objectives that are closely 
related by content and educational purpose. To form a 
segment, behavioral objectives similar in content were 
identified and sequenced. For examplQ, in Part Two, 
Individual Behavior, objectives based on ''motivation" 
were grouped into a single segment (Part Two, Segment VI) 
Likewise, objectives based on "conflict" were grouped 
into a single segment (Part Two, Segment VII). The 
segment served as the logistical unit in implementation 
for purposes of scheduling and assessment of progress 
through the course materials. .Depending on its purpose, 
a segment was classified as core, depth core, or 
enrichment. 

Core segments are required segments which include 
all of the information requisite to the attainment of 
behavioral objectives. They were presented sequentially 
and were designed for implementation in an individually 
paced instructional system. 



Depth" core segments are assoQiated with one or more 
segments and are directed toward amplifying the learning 
objectives of those segments. Depth core segments 
included in the first validation were film, group 
discussions, and classroom lectures by the USNA instructor. 
Unlike core segmer:tSj depth core were scheduled by the 
instructor with respect to time and place. 

Enrichment segments are related to but not essential 
to the mastery of terminal objectives. They were optional 
to students who desired more information than that 
presented in core segments. 
Ins true t ion a 1 Re se arcli 

■ Module. . A module is a particular instructional 
condition used to prepare and deliver materials for a 
segment, identified in terms of the categories of the 
Tosti and Ball (1969) model. Several parallel modules 
were prepared in each segment utilized for research 
purposes, representing variations specified by the 
experimental designs. The different modules of a 
segment are distinguishable from one another by differences 
in presentation design and/or media, . although the content 
is the same. " * " " 

Cumulative posttest unit . The cumulative posttest 
(CPT) unit is a group of three or four adjacent segments 
within a part. There are 13 CPT units involving 45 of the 
59- segments of the cour.sc. The primary criteria for 



.13 

grouping segments into CPT units were that the segments 
dealt with similar types of content and objectives, and 
that the instructional sequences relating to particular 
concepts which were initiated in the unit \i?ould also 
terminate in the same unit. All segments in a CPT unit 
were developed in the same medium and with the same 
variations in instructional conditions between modules. 

The outline of the course structure and media 
used is given in Appendix C. Hypothesis 1 was tested 
in CPT units 1, 3, 7 arid. 9, Hypothesis 2 \^as tested 
' in CPT units 4, 6 and 10, Hypothesis 3 was' tested in 
CPT units 5 and 13. Hypothesis 4 was tested in CPT 
unit 11. Section III of this report provides a brief 
overview of the various modules used within each of the 
media. These controlled_experiments and the variations 
in presentation design (modules) utilized in each are 
discussed in detail in the Report of Phase II Research 
Findings, Part 1: Conditions of Instruction (TR-6.12a). 
Test Organization 

Four different tests v;ere used throughout the 
course. They were the administrative pretest and posttest, 
the progress check, the cumulative posttest (CPT), and 
the USNA exajnination* 

The administrative pre and posttest was rn 80 point 
criterion referenced test composed of items representatively 
sampled from the objective-test item pool* There was at 



least one administrative test item for each segment of 
the course. The pretest was given at the beginning of 
the course, and the posttest was given as part of the 
final examination. 

The progress check was a criterion referenced 
test of approximately ten items. It was taken at the 
end of each segment. 

The cumulative posttest (CPT) was a norm referenced 
research test composed of positively discriminating 
content-related test items. Each CPT was composed" of 
ten items for each segment in the unit. Cumulative 
posttests were given at the end of each experimental 
unit. 

USNA examinations were a combination of criterion 
referenced test items selected from the objective-test 
item pool and items developed by the USNA on-site 
instructor. These were the only test results in the 
course which were used to determine the midshipmen's 
grades. 

IMPLEhJENTATlON 

The first implementation of the course was- conducted 
in the Spring of 1970. Forty-four USNA midshipmen 
participated. The course was administered by the USNA 
on-site instructor, the WLC on-site instructor, the V/LC 
systems analyst, and two data clerks. Specific procedures 



used in iinplcmonting the course are given in the 
Instructor's Guide (TR-6,6). 

The instructor's basic responsibilities were 
tutoring students needing remediation, leading group 
discussions I scheduling and administering depth core 
segments/ scheduling and administering examinations , 
and determining grades. 

The systems analyst developed and supervised 
the logistical procedures of the course. Data clerks 
controlled dissemination and collection of all core 
materials, remediation prescription forms, module 
questionnaires, progress checks, and cumulative 
posttests (CPTs). They alsj? scored progress checks 
and CPTs and forwarded data to WLC's computer center. 

Students were routed through the course according 
to procedures outlined in the S tudent Guide (TR-6.5). 
In brief, students worked through core segments of the 
course at their ov;n speed. They were allov/ed to check 
out software materials and study them v/henever and 
wherever they wished. All students v/ere given identical 
material when they studied a non-research segment; i.e., 
they were instructed by the same form of presentation. 
For research segments, they studied by the particular 
module (form of presentation) to which they v;ere assigned. 
Students were randomly assigned to modules at the 



beginning of the course, fiach student received his own 
routing schedule which included not only the sequence 
o£ segments he must study but also the schedule for 
remediation, research tests, and USKA examinations. 

Students wprlced through non -research material by 
studying a segment, talcing a progress check, remediating 
(if necessary), and tlien retaking the progress check. 
The requirement for remediation was based on failure 
to attain 80 t of the objectives as measured by the 
progress check. If the student failed to meet the 805 
• criterion on his first try, he was given a remediation 
prescription form which directed him to specific points 
in the content which related to the objectives failed. 
If the student failed to meet the 80% criterion 
following remediation, he reported to the on-site 
instructor for tutoring. 

Students worked through research segments in the 
same manner as non -research segments except that they 
did not remediate until after they had completed the 
entire research unit and taken the cumulative posttesi. 
A flow chart of the student's path through materials, 
progress checks, CPTs, and remediation is given in 
Appendix D. • 



FACILITIES 

For the implementation of the course , WLC was 
provided three classrooms at the Naval Academy. One 
roomi which was designated as the adm^inistrative office, 
containeil desks for the administrative staff and 
storage space for half of the course materials (including 
tapes » printed material, tests, forms, and computer 
cards). The administrative room was used as the site 
for administrative conferences, for student tutoring, 
and for distribution an.d collection of all material. 

The second room was used as the principal 
instruction room. It contained 15 student carrels 
equipped with Ampex VTR's (4900), TV monitors and 
earphones, and Crairg cassette recorders. 

The third room, used as a multi-purpose room, 
had three (fully equipped) carrels to handle overflow, 
from the instructional room. In addition, there were 
30 student writing desks which were used during depth 
core lectures, films, group discussion's, and testing. 



III. DE3CRIPTIC..' OF MHDIA AND PRESENTATION FOllIviS 

Media used in the first implementation of the USNA 
leadership course were audiotapes with panelbooks, 
videotapes, linear programed" texts, ^yndactic texts, 
audiotapes with intrinsically programed texts, and 
computers (see Appendix C). A description of the media 
and each of the variations in presentation form (modules) 
within each medium is given in this section. 

VIDEOTAPES 

Videotapes were used to present core content in 
lecture format. The lecturer for all videotape segments 
was an active Naval officer '.^ith teaching background. 
With the aid of a teleprompter , the lecturer presented 
all material verbatum from a prepared script. All 
videotapes were prepared at. a commercial television 
station. Commercial quality quad video tape recorders 
were used v/hich provided broad editing capability 
unavailable in one inch VTR format.. Two cameras were 
used in taping the lecture to allow for . integrati on of 
a series of visuals (charts, photographs, drawings, etc.). 
Additionally, key points of content v;cre superimposed oh 
the screen during the program. Four different modules 
x^cre prepared which involved variations in the response 
demand frequency (IIDF) and the response demand (RD) 
dimensions of the presentation design. 



.Module 1: (High RDF-covert response demand) 
Module 1 of the videotape lecture was characterized 
by high response demand frequency. At appropriate points 
in the lecture, the lecturer referred the students to a 
numbered question in the panelbook. Students read the 
question and recorded their answers in the panelbook. 
Sufficient time was allowed for students to respond so 
they did not have to turn off the VTR. The number of 
questions asked ranged from 15 to 22. Module" 4 differed 
from Module 1 only in that an overt response was required 
of the student. 

Module 2: (Low RDF- overt response demand) 
The low response demand module of the videotape 
lecture was developed by simply editing out all but 
three of the lecturer's references to questions in the 
panelbook. All other elements of the videotape lecture 
remained the same. Module 3 differed from Module 2 only 
in that a covert response was required of the student. 

AUDIOTAPES 

Audiotapes with panelbooks were used to present 
core content in lecture format. The lecturer was a 
commercial radio announcer. The lecturer presented all 
material verbatum from a prepared script . In all audio- 
video research segments, the scripts for audio and video 
presentations were identical, as were the modules employed* 



All audiotapes wore' developed in a commercial 
recording facility. . In developing the audio presentations, 
standard recording tapes v/ere used. For student use, 
tapes were transferred to C~60 and C-90 cassette 
cartridges . 

All charts, photographs, drawings, etc., accompany in 
the audiotape lecture were presented in a panclbook. 
(It should be noted that for videotape modules, these 
charts, etc., were presented on the VTR.) 

Module 1: (High RDF-covert response demand) 
Module 1 of the audiotape lecture was characterized 
by high .response demand frequency. At appropriate points 
in the lecture, the lecturer referred the students to a 
numbered question in the panelbook. Students read the 
questions and recorded their answers in the panelbook. 
Sufficient time was allowed for students to respond so 
they did not have to turn off the- cassette recorder. The 
number of questions asked ranged from 15 to 22. Module 4 
differed from Module 1 only in that an overt response 
was required of the "student. 

Module 2: (Low RDF-overt response demand) 
The low response demand module of audiotape 
presentations was developed by simply editing out all 
but three of the lecturer^s references, to questions in 
the panelbook. All other el em.ents of the audio lecture 



remained constant. Module 3 differed from Module 2 
only in that a covert response was demanded of the 
student. 

LEARNING ACTIVITY SUMMARY (LAS)- 

A learning activity summary is similar to a 
traditional textbook ot bibliography approach to 
learning. It is a technique very often used in college 
and graduate seminars to put the responsibility for 
structuring learning and achieving objectives on the 
student. A learning activity summary. was composed of 
three parts: 1) an overview of the segment, 2) behavioral 
objectives for the segment, 3) a bibliography of source 
material that was related to each of the objectives. 
Students worked through an LAS segment by reading the 
overview and objectives and studying text materials 
which were related to each objective. Text materials 
were either select pages in published text books or 
supplemental handouts. 

The student studied all text material until he felt 
he had mastered the objectives and could pass the progress 
check. If he did not achieve 80^ of the objectives on 
the progress check, he remediated by re -studying the 
prescribed text material. In Module 1 in this medium, 
pairs of students studied together. In Module 2 students 
studied individually. 



LIN13AU PROGRAMED TEXT 

Linear programed texts used in the first validation 
were developed by the RULEG and EGRUL methods o£ programing 
(Rule -example; example-rule). These are essentially- 
programing methods of presenting a rule (definition, 
principle) and having the student identify an example 
of the rule (from 2, 3, or 4 choices)," or presenting an 
example and having the student identify the rule or 
principle which is depicted in the example (Markle, 1964)'. 
Variations of the RULEG-EGRUL method which were used 
are EG-EG and RUL-RUL. (See Appendix B, Guidelines for 
Development of Behavioral Objectives.) 

It is important to note that although confirmation 
of responses is ordinarily an important part of programed 

■instruction, confirmations were deleted in the first 
implementation in order to obtain valid data on student 
frame responses. The first three modules of linear 
text were presented in standard format; i.e., there was 
a question for every information frame. This defined 
the high response demand frequency (HRDF) dimension. The 
only difference in presentation among these modules 

•was in the form of response required of the student. 
Module 1: (HRDF-written RD) 

Students were instructed to respond to each frame 
by writing their selection (A, B, C, or D) on the frame 
answer sheets. 



Module 2: (HRDF- spoken RD) 

Students were instructed to respond to each frame 
by speaking their selection (A, C, or D) into a 
tape recorder. 

Module 3: (HRDF-covert RD) 
■ Students were instructed to read each frame 
question and think the answers to themselves. 

Modules 4, 5, and 6 of the linear text covered 
the same, material as the first three modules, but the 
frequency of response demand varied within the 
presentations. Instead of asking a question fop: every 
frame, questions were asked for every second or third 
frame. In "no-question^^ frames, examples or principles, 
which would be deleted when questions were deleted, were 
reworded in statement form; e.g., instead of asking, 
"Which of these situations best exemplified principle X?," 
the frame was followed by a statement, such as "An 

example of principle X is " Modules 4, 5, and 6 

differed from each other in the form of response demand. 

Module 4: (LRDF-written RD) 

This was the same as Module 1 except for low 
response demand frequency (LRDF). 

Module 5: (LRDF- spoken RD) * 

This was the same as Module 2 except for low 
response demand frequency. 



Module 6: (LUDF-covcrt RD) 
This was the same as Module 3 except for low 
response demand frequency. 

SYNDACTIC TEXT 

A syndactic text is essentially a series of 
linear programed frames each preceded by a brief but 
complete summary of the information presented in the 
frames.. Students worked through the syndactic text by 
reading the first summary statement and taking a. 
summary quiz of 5 to 8 questions. If the student 
answered all summary quiz questions correctly, he read 
the second summary, took summary quiz 2, etc. 

The student who incorrectly answered one or more 
questions of a summary quiz was required to study the 
linear programed sequence associated with that summary. 
The linear programed sequence was identical to the 
linear text discussed on page 21. It was developed by 
the RULEG or EGRUL method of presenting small bits of 
information accompanied by examples of the concepts being 
taught. At the end of the programed sequence, the student 
retook the summary quiz. Regardless of his performance 
when he retook the summary quiz, he went on to the next 
summary statement and repeated the procedure. Non-research 
segments of syndactic text were implemented according to 
the procedure given above. 
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. Module 1: (HRDF) 
The first module of research segments of syndactic 
texts was presented in the "same manner as non-research 
segments. It was characterized by high response demand 
frequency(HRDF) • Module 1 is depicted in chart form as: 

PART TWO 
SEGMENT SIX 




Module 2: (LRDF) 

The second module of syndactic text was identical to 
Module 1 except .that it \v'as characterized by a Idv; response 
demand (LRDF) program. Instead of- having RULfeG question 
frames, examples \\'ere simply given in statement form. This 
sequence of statements \\'as referred to as a "detailed 
summary statement.'' There were no questions asked in the 



detailed summary statement'. Module 2 is depicted in 
chart form as; 

PART TWO . 
SEGMENT SIX 



^^ummar^^^^ 



Summary 
Quiz 1 
100% 



Yes 



No 




Detailed 
Summary 
Statement 1 
(LRDF) 



4 



Summary 
Quiz 2 
1001 



No 

4/ 



± 



Summary 
Quiz la 



Detailed 
Summary • 
Statement 2 
(LRDF) 




Yes, 

^ ^Summary 3 



Summary 
Quiz 2a 



Module. 3: .(no RDF) 

The third module was represented by summary statements 
alone (no remediation, no RDF). The student read a summary 
statement, took the summary quiz, and proceeded to the next 
summary statement regardless of his score on the summary 
quiz. 



Module .3 is depicted in chart form as: 

PART TWO 
SEGMENT SIX 




AUDIOTAPE Al^ID INTRINSICALLY PROGRAMED TEXT 

As originated by Norman A. Crov/der, ' the intrinsic 
programing technique consisted of routing a student through 
a "scrambled" text on the basis of his response. Each 
response directed him to a different page of the text; 
thus, the student could not read through directly and 
sequentially. 

. Combining the intrinsic programing technique with 
an audiotape was a WLC innovation. In this teaching mode, 
the information was presented via the tape. IVhile the 
student listened to it, he also looked at a summary page 
in the text which contained a precis of what he was hearing. 
He then stopped the tape and followed the instructions at 
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the bottom of the sunimary page, directing him to a page 
containing a question which tested the information given 
on the tape and summary. Each response to the test item 
referred the student to another page which informed him 
of the accuracy of that response,' Thus-, the student 
would select the alternative which he thought was correct p 
turn to the page indicated for that alternative, and find 
out if he had made a correct selection. If he had 
selected the correct response, he was instructed to go on 
to another summary page which he read while listening to 
the next audio portion. If his response was partially 
correct or incorrect, he was either told the nature of 
his error and instructed to proceed as described above, 
or he was instructed to return to the summary or question 
page to study the information again and select another 
alternative. This process of interaction between tape 
and text continued throughout the segment. 

The tape, which contained the content of the 
segment, remained the same throughout the four modules. 
The text differed as follows: 

Module 1: (HRDF-HMF) 

In Module 1 a question was asked for each informa- 
tional frame (HRDF) . Based on his response, 'the student 
was always branched to a page where his answer v;as 
discussed and confirmed or rejected (high management 
frequency) . 



Module 2: (IIRDP-MMF) 

A question was asked for each frame (IIIUDF) , but 
the student was branched on the basis o£ his response 
for only 501 o£ his responses (medium management frequency). 

Module 3: (IIRDF-LMF) 

A question was aslced for each frame (HRDF) , but 
the student was never given feedback nor confirmed; i.e., 
he was never branched as a result of his selection. He 
simply went on to the next question (low management ^ 
frequency) . 

Module. 4: (LRDF-LMF) 

Only three or four questions were aslced throughout 
the programed sequence (LRDF), and the student was not 
branched on the basis of his responses to the three or 
four .questions (low management frequency). 

COMPUTER ASSISTED . INSTRUCTION 

Part T\^elve (Applied Leadership) '/as developed as 
CAI material for the 1500 Instructional System. All of 
the three cpmponents of the system (CRT, audio, and image 
projector) were utilized in the implementation of the four 
modules. These modules paralleled the four modules used 
in the audiotape- IPB segments. 

Module 1: (HRDF-HMF) 

Module 1 was established in the follov/ing pattern: 
informational framo(s), questions, and feedback on the 
student's response. The informational frames v;ere 



presented on the CRT screen and image projector. The 
questions, which were often situations in which the 
student had to decide, the best course of action, were 
presented: 1) on the audio, where the situation was 
described; 2) on the image projector, where pictures of 
the situation were presented along with the audio; 
3) on the CRT screen, where the student was asked, to 
select an answer from 3 to 5 choices. The student's 
selection, accompanied by feedback, was displayed on 
cither the CRT screen or the audio tape, and occasionally 
.on the image projector. This feedback consisted of the 
reason (s) why the selected answer was correct or incorrect. 
Module 2: (HRDF-MMF) 

Module 2 was developed from Module 1. The 
difference between the modules was in the amount of 
feedback. In Module 2, feedback was administered to a 
student's response only after every other question. In 
other words, the student-was given a question, he 
responded, and he got feedback. He was then presented 
another question, he responded, and he got no feedback. 

Module 3: (HRDF-LMF) 

Module 3 was also based on Module 1. The student, 
however, got no feedback at all to any of his responses. 



Module 4; (LRDF-LMF.) 

Module 4 contained the same material as Module 1. 
However, the student was presented only three questions 
in the entire segment. Instead of the questions that 
were presented in Modules 1 through 3, the material was 
presented in statement form, for example, a situation 
was described on the audiotape with a picture displayed 
on the image projector. Instead of asking. the student 
a question pertaining to the situation, i,e,, "Wliat is 
the best way to handle the situation?," the student was 
presented a statement of the correct alternative to the 
question asked in the other modules, i,e,, "The following 
is the best way to handle the situa':ion." As in Module 3 
no feedback was given. 



IV. DESCRIPTION OF TESTS 

The basic tests used to collect performance data 
during the first implementation were administrative pretest 
and posttest, progress check, and cumulative posttest. 
Brief descriptions of the development of each of these 
tests and the summary text statistics collected during 
their development are presented in this section. For 
detailed descriptions, see Research § EvaTuation Plans , 
Part 1 (TR-6.3a). 

TEST ITEM POOL 

In order to ensure that at least the majority of 
course objectives were measured during the first validation, 
a test item pool of criterion referenced test items v/as 
developed. The original test item pool consisted of l,4i6 
criterion referenced items. The specification for 
developing test items was that two test items be developed 
for each of 12 objectives in a segment, i.e., 12x2x59=1,416. 
The selection of objectives to be included was based 
primarily on the need for representative coverage of 
terminal objectives, and secondarily on representative 
coverage of enabling objectives. All test items in the pool 
bore a one to one relationship to behavioral objectives 
and content. Since both WLC and USNA subject matter experts 
assisted in the development and reviev; of the test items. 
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content validity for all items was assumed. (An example 
o£ a behavioral objective and its corresponding criterion 
referenced test item is given in Appendix K.) Objectives 
not covered by test items in the test item pool were 
measured within the course materials by' criterion frames. 

ADMINISTRATIVE PRE AND POSTTEST 

Administrative tests were developed to provide a 
basis for evaluating total course achievement. The 
administrative pre and posttest was actually one 80-item 
test which was administered once at the beginning and 
once at the end of the course. (The administrative pre 
and posttest is distinct from USNA examinations which 
were administered throughout the course .by the USNA 
instructor for purposes of assigning four-week and 
final course grades.) 

•The administrative pre and posttest was developed 
by representatively selecting test items from the test 
item pool. In this way, there was at least one test item 
selected from each segment of the course, plus an 
additional test item from each of 21 segments. Validity 
assumed for the administrative test was a high degree 
of content validity based on three SMEs' approval of 
test itcm-objective-content interrelationships. Reliability 
for administrative tests was estimated by the Kuder Richardson 
formula 20. Reliability coefficients, means, and standard 
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deviations for the administrative, tests arc given in 
Table 4.1. 

TABLE 4.1 

RELIABILITY COEFFICIENTS, MEANS AND STANDARD 
DEVIATIONS FOR ADMINISTRATIVE TESTS 



Mean SD 

Pretest 40.2 5.34 

Posttest 55.6 4. 11 



It can be noted that the reliability for the pretest 
is greater than the reliability for the posttest. Perhaps 
the best explanation for the difference in pretest-posttest 
reliabilities is that while only 71 of the pretest item 
difficulties were 901 or higher, 271 or 361 of the item 
difficulties on the posttest were 90% or higher. The 
effect on reliability of having a test with many high 
difficulty items is the same as having a shorter test; with 
short tests, reduced reliability can usually be expected. 

Within the present course, lack of posttest 
reliability is not considered to be an indication of poor 
test construction. Since administrative tests were 
composed of criterion referenced items, it was hoped that, 
in fact, all students would answer all items correctly, 
in which case the reliability o£ the posttest would be 
zero. In other words, the more successful the course and 
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the better the performance on each item of the posttest, 
the les.s measured reliability can be expected. 

Frequency distributions of item difficulty and 
discrimination indices for the posttest are given in 
Appendix F. The item difficulty index is simply the 
percentage of students responding correctly to each 
item, i.e., the more student responses the higher the 
difficulty. The item discrimination index was computed 
by point bi-serial correlation of correct -incorrect 
responses and total test ranks for each item. 
Appendix G is an example of the type of calculations 
obtained for each item in computing the discrimination 
index. 

In addition to the frequency distributions 
(Appendix F) and the individual item discrimination 
statistics (Appendix G) , a concise summary of the item 
difficulty and discrimination index along v^ith its 
significance for each item in the text was provided for 
the writer and analyst. This test item statistics summary 
for the posttest is given in Appendix H. Discrimination 
indices reported in Appendix H ranged from -^.79 to .88. 
However, the wide range of discrimination is misleading 
because if the difficulty index for one item is either very 
high or very low, the discrimination index can be seriously 
affected by one high scoring student *s incorrect response 



or one low scoring student's correct response, i,e,, 
the more uniform all students' responses to items, the 
more sensitive the discrimination index is to variations • 
among the few remaining individuals. 

It can be noted in Appendix H that although there 
is a wide range of discriminating items, only two 
discrimination indices were statistically significant. 
Since administrative tests are composed of criterion 
referenced items, it is not desirable to have many items 
which discriminate positively among students, and 
certainly undesirable to have items which discriminate 
negatively among students. In revising items for future 
implementations, more consideration was given to the 
•selection of good growth items than to the selection 
of items which would discriminate among students • That 
is, items were selected on the basis of their ability* 
to yield low difficulty on the pretest and high difficulty 
on the posttest. 

In addition to item difficulty and item discrimination 
an item response analysis was made of each administrative 
pretest and posttest question (see the example in Appendix I) 
From this analysis, the writers and analysts were able to 
determine the strength of item distractors as well as the 
general difficulty of the item. It was felt that the use 
of the ratings made by the, students of the confidence 



they had that the response they selected was correct 
would be of aid in revising the items. For example;* 
i£ most of the students got an item correct but had 
very low confidence, the item would be reviewed for 
possible change. It was found, however, that most 
students responded with a rather high level of 
confidence in most situations. Therefore, confidence 
ratings associated with item responses were generally 
of limited value in revision. 

PROGRESS CHECKS 

Progress checks were developed to measure student 
achievement on each of the core segments. They v/ere 
ten-item tests made up of criterion referenced items 
drawn from the test item pool. Ten items for each of 
the 59 core segments were drawn from among 24 items 
originally developed for each segment. . Since they were 
dravai from the test item pool, which had met SME 
approval, content validity for progress checks was 
assumed. (No estimate of the reliability of progress 
checks is reported because the tests were composed oP. 
only ten criterion referenced items.) 

Frequency distributions of item difficulty and 
item discrimination for all progress check items are 
given in Appendices J and K. It can be noted that 
most of the discrimination indices approach zero. High 
positive or negative discrimination indices are related 



to Very high or very low difficulty items (as is the 
case with administrative items). A general rule of 
thumb is: if the difficulty level of the item is below 
40% or above 80%, the discrimination index com.puted by 
the item- total correlation method is questionable. 

Analysis was made of all responses to each progress 
check item. An example of the format for item analysis 
is given in Appendix I. 

CUMULATIVE POSTTESTS 

Cumulative posttests were developed to assess the . 
effects of variations in presentation across research 
segments. They were either 30- or 40-point tests 
designed to have high positive discrimination-low 
difficulty test items. CPT units are. related to course 
content across segments in research units, rather than 
being directly related to terminal or enabling objectives. 

Cumulative posttests are* characterized by two 
types of test items: 1) simple associations, such as 
definitions and identifications, and 2) complex problem 
solutions, such as EGRULS. Validity for CPTs was assumed 
on the basis of representative sampling of content. 
There \^as considerable input to test item development 
by subject matter experts, although no attempt was made 
to have three independent judges review and signoff the 
test items. Reliability coefficients for CPTs were 
computed by Kuder Richardson-20 (Table 4.2). 



TABLE 4.2 

* 

RELIABILITY COEFFICIENTS FOR CUMULATIVE POSTTESTS 



Un it No. No," of items ■ ■ • • y ■ 

1 40 * 

2 • 30 * 

3 4:. 40 * 

4 30 * 

5 40 * 

6 . 30 .64 

7 40 .68 

8 30 • .58 

9 40 .67 

10 30 .18 

11 30 .69 

12 .30 .65 

13 , 40 .58 

* Coefficients for those units were in error at the 
time of this reporting and are being re-computed. 
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As shown in Table 4:2, reliabilities for CPTs 
were reasonably high, considering the shortness of 
the tests. It is interesting to note that in contrast 
to pdministrative tests, high reliabilities for CPTs. 
are desirable features of the tests. CPTs are designed 
for medium difficulty and positive discrimination 
among students and, as such, should show more item- 
item, item-total correlation than administrative tests. 
In revising CPT items, special attention was given to 
the discrimination level of CPT items as well as to 
item difficulty and item response distributions. 
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V. RESULTS OF TOTAL COURSE EFFKCTIYHNnSS 

Since the Leadership Management course is based on 
a behavioral approach to instruction, the first validation 
of total course effectiveness vas based primarily on 
student test performance over a series of behavioral 
objectives. The premise for this form of evaluation is 
that once behavioral objectives are developed for a course 
and everyone agrees that they are necessary and worthwhile 
objectives, then the test of the effectiveness of the 
system is simply whether students attain the objectives. 

The measurement of student performance on stated 
behavioral objectives is a technique for assessing the 
absolute effectiveness of a system. Relative effectiveness 
is assessed by comparing the effectiveness of one system 
or portions of the system to ocher systems. The relative 
effectiveness of the USNA multi-media system to one or 
more of the existing USNA leadership courses has not been 
assessed for the following reasons: 

1. The evaluation of the multi-media course is 
based on student performance on over 600 behavioral 
objectives. A comparison of the effectiveness of the 
multi -media course to other courses would therefore 
necessitate the inclusion of over 600 measured objectiv^es 
in the other courses. To compare* effectiveness based on 



final examinations alone would mean comparing effectiveness 
on only a sample of objectives rather than total objectives, 

2. A second consideration in multi -media vs. 
existing leadership course comparisons is the possible 
Hawthorne and Rosenthal effects which may bias results. 
These two effects are, respectively, the tendencies 
X) for students to realize they are in' an experiment and 
to perform beyond typical expectations (Schramm, 1964) , 
and 2) for teachers to realize the*y are being compared ' 
and thus alter their typical patterns of instruction 
(Rosenthal, 1966). 

3* Assuming that effectiveness data for the multi- 
media course and the existing leadership course were not 
identical, there would be no way of accounting for the 
differenc6s. Since several different media and forms of 
presentation are being used in the USNA course, and since 
teaching methods and materials vary from one USNA 
instructor to another, there would be no clear cut 
indication of the conditions of instruction which account 
for total differences. 

4* Within the multi-media system, the effort is 
made to compare the relative effectiveness of one mode 
of presentation to another. Included in the .forms of 
presentation employed are lectures presented via audiotape, 
textbooks, and bibliographies, all of which are traditional 
instructional techniques. In making these comparisons. 



all variables except the relevant presentation variables 
are controlled; i^e,, stiulcrits are given the same content, 
objectives, materials, and test items • Only under these 
circumstances is it possible to state that one form of 
presentation is relatively more effective than another* 

5, The multi -media course is not intended to 
supplant the instructor in other courses. It is intended 
rather to teach effectively the core content requisite 
to the understanding of Naval leadership, thereby reducing 
the need for the instructor's role as strictly a 
disseminator of information. An instructor using multi- 
media materials need only augmen*: prepared materials with 
personal guidance of students. That is, he is able to 
lecture on points he would like to highlight, lead group 
•discussions, tutor and counsel students, and in general 
use his time as a professional to invent new and creative 
ways of simulating leadership experiences. V/ith these 
considerations in mind, studies of the effectiveness of 
systems which essentially compare one role of the instructor 
to another role are too global to be of value. 
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TOTAL COURSE EFFECTIVliNESS 

As mentioned in previous sections, the evaluation of 
course effectiveness has been based on student performance 
1 over two types of criterion referenced tests: the 

administrative pre and posttest and the progress check. 
Since both administrative and progress check tests 
consisted of items which had one-to-one correspondence 
with behavioral objectives, it is reasonable to estimate 
the average percentage of objective attainment by the 
average percentage of test items answered correctly* To 
the extent that test items and objectives have been 
directly matched and are thoroughly represented throughout 
the course, the percentage of objectives attainted and the 
percentage of test items achieved can be viewed inter- 
changeably. 

Results of total course effectiveness based on 
administrative tests represent Estimates of effectiveness 
from a sample of objectives. Results of total course 
effectiveness based on progress checks represent estimates, 
of effectiveness over all course objectives. Evaluation 
! methodologies and results, based on both types of tests 

are given in this section* 

TOTAL COURSE EFFECTIVENESS BASED ON ADMINISTRATIVE TESTS 

As discussed in Section IV, administrative pre and 
posttest was actually the same 80 point tost given at the 



beginning and end of the course. It consisted of items 
representatively sampled from each of the S9 core 
segments. It should be noted at this point that five 
items on the pretest were judged to be ambiguous by the 
WLC and USNA on-site instructors after the pretest had 
been administered. Since those five items were 
replaced for the posttest administration, both the 
.five pretest items and the five replacements were 
dropped' f rem the analysis of total test performance. 
Analysis was therefore based on 75 pre and posttest 
items rather than 80. _ 

The most direct but somewhat misleading estimate 
of total course effectiveness is found in the average 
percentage of test items answered correctly on the 
administrative posttest, Wi)ich is 74.lt*. This estimate 
is misleading because: 1) it represents only the average 
percentage of objectives covered on the posttest 
(75 objectives), and 2) it does not take into account 
how much the students knew when they first entered the 
course (Deterline, 6Cth Yearbook NSSE , 1967). In light 
of this problem, to obtain a more comprehensive 
estimate of total course effectiveness, it is necessary 
to: 1) consider performance on all objectives measured 
throughout the course, and 2) consider the entering level 
of knowledge of the students. (Performance measures on 
all course objecti\'es are given in Section VI •) 



In order to assess the students' entering 
familiarity with the content, all students were pretested 
on the administrative test. The average percentage of 
pretest items achieved was 53.6%, and the average 
difference in amount achieved from pretest to posttest 
was 20.5% (74.1 - 53.6). (See Appendix L for the 
frequency distribution of pre and npsctest scores.) 
The actual percentage gained from pretest to posttest 
is still a relatively obscure index of course effectiveness 
because it does not compare the amount of gain that was 
possible. 

An index which does represent how much students 
learned from the system with respect to how much they could 
have learned is the ratio of actual gain to maximum 
possible gain. (Posttest minus pretest divided by -the 
number of points on the test minus pretest.) The 
average crude gain ratio (raw spore gain .ratio) for the 
first implementation was 44.3% (Table 5.1). This 
indicates that, on the average, students learned approxi-. 
mately 44.3% of what they could have learned. 
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TABLE 5.1 



MEAN GAIN RATIOS BASED ON ADMINISTRATIVE PRE AND POSTTEST 







Mean 
■ pretest 


Mean 
■ pes tt est 


Mean 
gain ■ 


Maximum 
possible 
■ gain 


Gain 
■ ratio 


Raw (crude) Score 


40.2 


55.6 


15.4 


34.8 


44.3 


True Score 


40.1 


55.5 


15.4 


34.9 


44^ 


N = 75 













One problem associated with the use of the crude gain 
ratio is that it does not take into account the lack o£ 
reliability o£ either the pretest or posttest. Lack of test 
reliability has the effect of allowing varying results on 
the same test for the same students. For example, if a test 
were perfectly reliable, repeated testings of the same 
student would yield the same test scores. Proportionate 
to its lack of reliability, an unreliable test would produce 
varying test scores for the same student. The effect of 
test unreliability on the gain score would be that a 
.student *s score could vary unwards from his true score on 



the pretest and vary downwards from his true score on the 
posttest, or vice versa. In either case, results based 
on the difference between pre and post testings of 
unreliable tests would yield unreliable data since it 
would indicate very small gain in the first case and 
large gain in the second. 

In order to account for the lack of reliability 
of pre -and posttests in computing the gain score ratio, 
estimates of the true gain scoz*e ratio for all students 
was computed- (McNemar, 1958, and Ilorst, 1966). 

Results in Table 5.1 indicate that the average 
estimated true gain and the true gain ratio, when correcte 
for test reliability, are virtually identical to the 
average crude gain and the crude gain ratio. Hov/ever, 
it can be noted from Table 5.2 that the estimated 
true gain ratios for individual students are far more, 
homogeneous than the crude gain, ratios for individual 
students. (See also Appendix M for differences in true 
and crude pretest scores and true and crude gain for 
individual students.) 



TABLH 5.2 

FREQUENCY DISTRIBUTION OF GAIN RATIOS 



Crude gain Estimated true 
^^tio. frequency. gain .fre.q.uency 

65.0-69.9 1 

60.0-64.9 3 

55.0-59.9 4 

50.0-54.9 8 

45.0-49.9 4 13 

40.0-44.9 5 31 

35.0-39.9 9 

30.0-34.9 ' 5 • 

25.0-29.9 3 

20.0-24.9 ' 2 

15.0-19.9 



A second problem associated with the use of gain 
score ratios in general is that there is no established 
standard by which to gauge them. To indicate that a 
system is 441 effective may seem to indicate that the 
system is not operating at an acceptable level of 
effectiveness. However, when the gain scor^ ratio is 
interpreted in conjunction with the posttest average, 
it becomes more meaningful. Considering' both the true 
gain score ratio and posttest average, it can be noted 
that students gained /;.nly 441 of what they could have 
gained, but their final scores were fairly high, i.e., 74 

To further interpret the gain ratio and average 
posttest performance, it is necessary to consider each 
i'tpm of the test from which the data are derived. An 
additional element of data, therefore, is the test item- 
objective reference for the administrative pre and post- 
. tests which indicates the number of objectives (test 
items) most students knew prior to taking the course and 
the number of objectives students failed to achieve 
on completing the course (Appendices* N and 0). 

Refering to Table S.3 (Frequency distribution of 
number of objectives achieved by I of students), it can 
be noted that there were no objectives which all students 
knew upon entering the course. However, by combining 
the first three frequency intervals, it can be seen that 



805; or more students knew 17 of the items, or 23% of the 
7S objectives on the pretest. In these cases, it is easy 
to understand that there was little room for gain on 
these items. On the other hand, although there were six 
items achieved by 100% of the students, by combining 
the last five frequency intervals for the posttest 
(Table 5. 3), it can be noted that there were 11 items 
that were achieved by 501 or less of the students. 
This also accounts for low gain. From Table 5.4 it can 
be noted that there were additionally eight items which 
produced negative gain from pretest to posttest 
administrations; i.e., the students scored lower on 
the posttest than they did on the pretest. The use 
of the type of data discussed thus far in revising 
the course in outlined in Section VIII. 
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TABLE 5.3 



FREQUENCY DISTRIBUTION OF NUjMBER OF OBJECTIVES ACHIEVED 
BY PERCENTAGE OF STUDENTS: ADMINISTRATIVE TESTS 



Percentage of 


Pretest ■ 


Posttest 


students 


number of 


number of 


achieving 


objectives 


objectives 


objectives 


achieved 


achieved 





100- 


0 


6 


90-99 


5 


21 


80-89 


12 


9 


70-79 


7 


13 


60-69 


8 


8 


51-59 


10 


7 


40-49 


8 ■ 


3 


30-39 


8 

1 


4 


20-29 


11 


1 


10-19 


3 

• 


3 


0-9 


3 


0 



TABLE 5.4 



* 

FREQUENCY DISTRIBUTION OF PERCENTAGE OF GAIN 
OF NUMBER OF OBJECTIVES FROM PRE TO POSTTEST 



Percentage ' Number of 

■ -^^^^ .O.b.j.e.ct.i.v.es 



80 to 89 


1 


70 to 79 




60 to 69 


5 


50 to 59 


3 


40 to 49 


7 


30 to 39 


7 


20 to 29 


10 


10 to 19 


15 


0 to 9 


19 


-9 to 0 


2 


19 tp-lO 


4 


29 to-20 


1 


39 to-30 


1 



Results from the data based on the administrative 
test item-objective reference (Appendix N and 0) indicated 
two points quite clearly. One was that although students 
achieved an average of 74.11 on the posttest, a number 
of the tost items were not adequate and needed to be 
revised. ' The second was that the objectives covered 
by the administrative test needed to be re-examined with 
respect to the adequacy with which they were taught • 
The percentage of correct student responses to items 
on the administrative tests is identical to the item 
difficulty level for these items. Procedures for 
revising the administrative exam based on the test item- 
objective reference are expanded upon in Section VIII • 

TOTAL COURSE PERFORMANCE BASED ON PROGRESS CHECKS 

As discussed in the preceding sections, er»timates. 
of total course effectiveness based on administrative 
tests represent student performance over only a sample 
of course objectives* Since administrative tests sampled 
only 75 of the course objectives, test results do not 
give a clear picture of student performance on each 
objective measured throughout the course which is needed 
to properly revise and improve the course. 

In order to obtain data on all course objectives, 
progress checks were given at the end of each segment; 
they covered most of the terminal objectives and enabling 
objectives taught in those segments. (All enabling 



objectives not covered by progress checks v/cre covered 
by criterion frames within the program sequences.) 
Records of progress check results were mac'e for each 
student, and the average progress check performance 
was computed for all students • The exact score on each 
progress check for each student was included in the 
student track. (An example of the student track is 
given in Appendix P.) 

Since students took the progress check immediately 
after completing a core segment and then remediated on 
the basis of failure to attain 80% of the objectives, 
it is necessary to report progress check performance 
both before remediation (iteration i) and after 
remediation (iteration 2). The percentage of total 
objectives attained by each student and the mean 
percent across students both before and after remediation 
are presented in Appendix Q. The frequency distribution 
of the percentage of total objectives attained by 
number of students (Table 5.5) is compiled from 
Appendix Q. 



TABLH 5.5 

FREQUENCY DISTRIBUTION OF PERCENTAGE OF TOTAL COURSE 
OBJECTIVES ACHIEVED BY NUMBER OF STUDENTS: " PROGRESS CHECKS 



I total Number of students * Number of students 

objectives before remediation after remediation 

achieved 

90.0-94.9 1 
85.0-89.9 18 
80.0-84.9 16 21 

75.0-79.9 16 .4 

70.0-74.9 11 
65.0-69.9 1 



From Appendix Q it can be noted that the average 
percentage of objectives attained throughout the course 
prior to remediation was 77.7%. Following remediation, 
the average percentage of objectives attained increased 
to 83. 81* (These mean percentage figures v;ere obtained by 

a 

where a = the number of objectives attained by all 

students, b = the number of progress check items given 

throughout the course (530), and n the number of 

students in the course (44).) 

As an indication of the variability of percentages 

of progress checks, the following groupings into 

percent-percent performance are reported: 

Before remediation, 9 7.81 or more of the 
students achieved 701 or more of the objectives. 

Following remediation, 911 or more of the 
students achieved 80% or more of the. objectives. 

These figures appear to indicate that, on the average, 

the course is performing quite well; however, the same 

problem in interpreting administrative posttest results 

is present in interpreting progress check results. That 

is, although 901 or more of the students are achieving 

801 of the objectives, there are still some students 

performing below criterion level 'on some objectives. In 



the first case, it can be noted from Table 5,5, that only 

four s^tudents failed to achieve the 801 criterion level 

after remediation, and their scores ranged from 76% to 79%. 

However, the second point (failure to achieve an average 

of 16% of the objectives after remediation) is more 

critical since this could represent up to 93 objectives 

throughout the course (16% of 530), Recall that the mean 

percent of objectives achieved after remediation (as 

sho\m in Appendix Q) was 83.8 which leaves roughly 16% 

of objectives not achieved, 

■Although this may seem high, it should be remembered 

that if a studrnt did not achieve the objective after 

remediation, he received tutoring from the instructor. 

The revision process would be directed toward reducing 

the need for tutoring. 
« 

In order to determine if failure to attain 100% 
of the objectives was attributed to random errors on 
objectives or to certain poorly presented or poorly 
tested objectives, a test item-objective reference was 
developed for all progress check items (Appendix R) . 
From the test item-objective reference, frequency 
distributions of the percentage of students responding 
correctly to each test item before and after remediation 
were developed. 



By combining the first five frequency intervals in 
Table 5.6, it can be seen that 301 or 575 of the items 
were achieved by more than 80% of the students before 
remediatic" Conversely, 431 of the objectives were not 
achieved by less than 201 of the students. These results 
ihdicatt^d a need for revision of either 431 of the test 
items or corresponding materials or both. In order to 
localize whether materials or test items should be 
revised, the percentage of correct student responses 
to each test item following remediation was tablulated. 
Again by combining the first five frequency intervals, 
it can be noted from Table 5.6 that, following remediation 
404 or 761 of the test items were answered correctly by 
801 or more of the students. Implications of these result 
for revision were that if students remediated through 
the same materials and were unable to answer the test item 
the problem could still be either the test item or the 
materials. Oh the other hand, if they remediated through 
different materials and were ab3e to answer the test itsm 
correctly, the item itself was probably adequate and the 
core materials should be revised. (Details of the use of 
progress check data in revision of materials is given in 
Section VIII.) 
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TABLE 5.6 

FREQUENCY DISTRIBUTION- OF NUMBER OF OBJECTIVES ACHIEVED 
.BY PERCENTAGE OF STUDENTS: PROGRESS CHECKS 



No. of objectives No. of objectives 
Percentage achieved before achieved after 

• of students .remediation . .remediation .... 



100 .0- 


69 


121 


95.0-99.9 


80 


- 107 


90.0-94.9 


67 


82 


85.0-89.9 


48 


46 


80.0-84.9 


37 


48 


75.0-79.9 


56 


33 


70.0-74.9 


30 


14 


65.0-69.9 


27 


11 


60.0-64.9 


9 


16 


55.0-59.9 


12 


13 


50.0-54.9 


17 


12 


45.0-49.9 


16 


3 


40.0-44.9 


8 


5 


35.0-39.9 


10 


4 


30.0-34.9 


8 


5 


25.0-29.9 


12 


3 


20.0-24.9 


5 


4 


15.0-19.9 


6 


1 


10.0-14.9 


5 


'1 


5.0- 9.9 


5 


1 


0.0- 4.9 


3 





VI. PART AND SEGMENT EI-FECTIVliNESS 

Integral to the evaluation of total course 
effectiveness is the evaluation of independent parts 
and segments of the course. By assessing the 
effectiveness level of parts, and segments within parts, 
. it is possible to identify and revise those segments 
which are not effective, thereby increasing total course 
effectiveness in future implementations. 

EVALUATION OF PART EFFECTIVENESS 

Evaluation of both parts and segments has been 
based primarily on student performance on progress 
checks before remediation. The total evaluation for 
parts is based on the average percentage of progress 
check items achieved, the average amount of time tajcen 
by students to complete each segm.ent, and the percentage 
of students who indicated high or low interest in the 
materials before remediating. Summary results of 
analyses by part are given in Table 6.1. Student interest 
and time was tabulated from the module questionnaire 
shown in Appendix S. The students filled this out after • 
each segment. 



TABLE 6.1 

UNWEIGHTED MEAN PERCENTAGE OF PROGRESS CHECK ITEMS CORRECT, 
STUDENT TIME, AND PERCENTAGE ' OF STUDENT INTEREST VOR ALL PARTS 



■ 

Parts 


Avg. % 
PC items 
correct 


Avg. 

■ student 
time (min.) 


% student 

interest 

(high) 


1 student 

interest 

(low) 


Part One 


74.4 


60.9 


56.4 


8.6 


Part T\iro 


.70.6 


33.9 


54 9 


11 n 


Part Three 


73.9 


30.8 


36.0 


19.4 


Part Four 


82.8 


44.5 


.44.6 


14.1 


Part Five 


73.3 


45.8 


22.4 


23.8 


Part Six 


85.0 


45. 1 


.41.9 


5.9 


Part Seven 


76.1 


39. b 


29.7 


21.3 


Part Eight 


80.9 


57.4 


32.3 


16.8 


Part Nine 


84.0 


40.8 


37.7. 


4.7 


Part Ten 


91.0 


1 

48.5 


37.9 


7.5 


Part Eleven 


84.6 


31.5 


31.4 


6.1 


Part Twelve 


87.9 


54.4 


61.0 


3.5 




It should be noted that results by part are 
expressed in terms of the unweighted mean percentage 
of progress check items and student interest. This 
indicates that all data reported for a part represent 
the typical performance of segments within that part. 
For example. Part Two was characterized by segments 
in which l)the typical performance on objectives was 
70.61; 2) the average amount of time (reported by 
students) per segment was 33.9 minutes; and 3) the 
typical reaction of students to segments was: 
54.91 of the students reported the material as 
•interesting, and only 111 reported it as uninteresting. 

There are several reasons for variations among 
recorded data for parts. Some of the reasons are that 
data are averaged over differences in content, test 
items, media, and presentation forms. Differences can 
also be explained in terms of varying styles of writing 
among materials developers. 

EVALUATION OF SEGMENT EFFECTIVENESS 

Differences in student performance by segment are 
given in Appendix T. It should be noted that many of the 
same problems in interpreting differences between parts 
are also problems in interpreting ' differences between 
segments; i.e., segments covor different content, arc 
tested by different progress check items, and are taught 



by. different media and forms of presentation. Even so 
it can be observed from Table 6.2 that, for whatever 
reasons, performance on some segments is considerably- 
better than others. 



TABLE 6.2 



FREQUENCY DISTRIBUTION OF AVERAGE PERCENTAGE OF 
PROGRESS CHECK SCORES FOR ALL SEGMENTS 



Average Number 
percentage of 
correct "segments 



90.0-94.9 


7 


85.0-89.9 


9 


80.0-84.9 


13 


75.0-79.9 


11 


70.0-74.9 


7 


65.0-6r.9 


10 


60.0-64.9 


1 


55.0-59.9 


1 



It can be noted, for example, that in 29 of the 
segmen'ts students achieved an average of 80% or more 
of the objectives before remediation. In 12 of the 
segments, students achieved an average of less than 
70% of the objectives. The data therefore indicate ' 
the need for substantial revision to either the test 
items or materials in 12 out of 59 or approximately 
20% of the total core segments. (See Section VIII.) 
The test item-^ob jective reference is used for 
identifying the test items, objectives, or content 
which require revision in these segments (Appendix R) . 

Table 6.3 presents the frequency distribution 
of average student time by segment. The average 
amount of time spent on *each of the core segments 
ranged from 20 to 90 minutes.. 
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TABLE 6.3 



FREQUENCY DISTRIBUTION OF AVERAGE STUDENT TIME FOR ALL STUDENTS 



Time in 
minutes 



Number of 
Segments 



90.0 
85.0 
80.0 
75.0 
70.0 
65.0 
60.0 
55. 0 
50.0 
45.0 
40.0 
35.0 
30.0 
25.0 
20.0 
15.0 



■94.9 
■89.9 
■84.9 
-79.9 
■74.9 
-69.9 
-64.9 
-59.9 
-54.9 
-49.9 
■44.9 
-39.9 
■29.9 
■29.9 
■24.9 
■19.9 



1 
1 
2 
6 



0 



5 
11 
10 
10 
5 
1 
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It can be noted that although the range is 20 to 
90 minutes, 931 of the segments had average times 
ranging from 25 to 65 minutes, or roughly from half an 
hour to an hour. Segments not falling within the half- 
hour to hour average time frame' were ^eviscd with respect 
to projected estimates of student time required. It is 
important to remember, however, that the data reported 
on time are based on student self reports of the amount 
of time spent per segment prior to remediation. In 
other words,. more time would be required for remediating 
if test performance on segments were not at criterion 
level. Therefore, more consideration was given to initial 
student test performance during revision than to time 
spent. 

Student attitudes toward segment material were 
compiled from the module questionnaires. The first item 
on the questionnaire required the student to indicate on 
a five point rating scale whether his interest in the 
material was very much above average, above average, average 
below average, or very much below average. The data were 
tabulated by collapsing the above average categpries into 
percentage of high interest and collapsing the below 
average categories into percentage of low interest. The 
frequency distribution of percentages of students* high 
and low interest by segment are given in Table 6.4. 



TABLE 6.4 

FREQUENCY DISTRIBUTION OF PERCENTAGE OF 
STUDENT INTEREST FOR ALL SEGMENTS 





Percentage 
of student 
. . . . interest 


High 
. fr.eq.uency. . . 


Low 

. . . . frequency . 



80.0-84.9 


1 




75.0-79.9 


• 


• 


70.0-74.9 


2 


• 


65.0-69.9 


2 




60.0-64.9 


5 




55.0-59.9 


3 




50.0-54.9 


3 




45.0-49.9 


1 




40.0-44.9 


9. . 




35.0-39.9 


6 


3 


30.0-34.9 


9 


2 


25.0-29.9 


8 


3 


20.0-24.9 


.5 


10 


15.0-19.9 


3 


8 


10.0-14.9 




8 


5.0- 9.9 


1 


12 


0.0- 4.9 


1 


13 
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It can be noted that many more segracnts v/ere 
judged to be of high interest than of low interest. 
On the other hand, there were 18 segments in which 
20% - 405 of the students indicated low interest • 
Interpretation of segments with either high or low 
interest should be tempered with the realization 
that students' attitudes toward material may represent 
attitudes toward the content, toward the medium, 
tow rd the form of presentation or toward the quality 
of ti. presentation of the material • It can be noted 
from Appendix T that students generally reacted favorably 
to segments in which they also performed well. Whether 
students performed well because they liked the material, 
or whether they liked the mate.rial because they performed 
well is not clear. In either case, the revision of 
materials was guided primarily by students* performance 
on objectives, and secondarily by the average student 
time and student interest. 
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VII. SEGMENTS-WITHIN-MIiDIA KFPECTIVENESS ' 

As discussed in the previous section, there are 
problems with the interpretation of differences in part 
effectiveness because results may reflect differences 
in content, test items, media, or presentation forms. 
In an attempt to localize at least one of the 
variables which may account for differential 
effectiveness, segments taught by the same media 
have been grouped and analyzed. 

At the outset, it is important to mention that 
although segments have been grouped on the basis of 
media, the results indicated in this section shorld 
not be construed as evidence for the superiority or 
inferiority of one medium to another. These results . 
do not reflect inherent qualities of the media as 
such, but are rather- indications of the effectiveness 
of the materials which were developed for and 
presented by each medium. For example, it could be 
that videotapes as mode of transmitting X type of 
ini nation in course Y is the best possible medium 
which could be selected; however, according to the 
way they are used in the multi-media course, video- 
tapes may be less effective than other forms of 
presentation. The reason for grouping and reporting 



segment results by media is to localize the variations 
in effectiveness of segments which may be attributable 
to teaching via different media or forms of presentation. 
Results reported in this section are simply the average 
results obtained for all segments taught by a single 
medium throughout the course. The results do not indicate 

t 

comparisons of media made over identical content with 
identical test items. Results of segment effectiveness 
by media are given in Table 7.1. 

TABLE 7.1 

UNWEIGHTED MEAN I PROGRESS CHECK ITEMS CORRECT, STUDEiMT 
TIME, AND % STUDENT INTEREST FOR ALL MEDIA 



% Progress Average % Student I Student 



Check student interest interest 

Media items time (min.) (high) (low) 

CAI 87.9 .54.4 61.0 3.5 

LAS • 73.8 57.1 30.3 21.5 

Syndactic text 80.2 ' 43.2 47.5 10.6 

Videotape 73.5 43.2 31.1 17.3 

Audiotape 74.8 43.8 30.7 21.3 

Audiotape IP 84.4 47.7 44.7 10.2 

Linear Text 78.7 48.5 32.1 21.1 



It can be noted that, on the average, the most to 
the least effective segments were those presented by: 
1) CAI, 2) audiotape-IP booklets, 3) syndactic text, 
4) linear texts, 5) audiotape lectures, 6) learning 
activities summaries, and 7) videotapes. Although there 
are not largt differences between audiotape, videotape., 
and LAS segments, there are larger differences between^ 
these segments and CAI segments. 

Correspondingly, it can be noted that student 
interest reports favored segments -within-media in 
approximately the same rank order as student performance; 
i.e., CAI, audiotape- IP, and syndactic text were highest 
in student interest, and audiotapes, videotapes, and LAS 
were lowest in student intei-est, ihe rank order 
correlation between student high interest and performance 
across segments-within-media is .86. The rank order . 
correlation between low student, interest and performance 
is - .78. 

These results tend to indicate that, on the average, 
content presented in CAI, audiotape-lP, and syndactic' 
text in this course is both more comprehendable and more 
interesting to students. Before generalizing these results 
to all segments within these media, .it is necessary to 
remember that there are, of course, variations within all 
of the media; i.e., each group of segments -within -media 
has both reasonably effective and ineffective segments. 
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The unweighted mean percentage of performance, 
time, and student interest for both research and non- 
research segments within, each media is given in 
Appendix U. It can be noted that non-research segments 
of videotape are the only non-research segments which 
vary appreciably from research segments-within-media* 
For videotapes, the non -research segments wore ^lOmcwhat 
.higher (8.7% higher) • Implications of these results are 
that students were not penalized by experimental pre- 
sentations of material since the research segments on 
the average were presented just as effectively and 
interestingly as were non-research segments -within -media* 

A second method of presenting data on segments- 
within-uedia effectiveness is the percent - percent 
criterion. The percent - percent criterion is essentially 
a method of reporting the variability of the distribution 
of student scores by selecting a criterion point in the 
distribution of scores and reporting the percentage of 
students who achieved scores equal to or greater than 
the criterion (Appendix V) . 

For purposes pf the first implementation, two criterion 
score points were selected: 70% of the objectives prior 
to remediation and 801 of the objectives prior to remediation. 
Additionally, results are reported for the criterion: 
801 of the objectives after remediation- Reported in 
Appendix V are the percent - percent, data for all segments- 
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within -media* Table 7.2 presents a summary of the data 

in Appendix V for media* 
« 

TABLE 7.2 

UNWI3IGHTED MEM I STUDENTS SCORING 70% OR 805 ON PROGRESS 
CHECKS BEPORE AND AFTER REMEDIATION: SEGMENTS -WTTMIN-MEDIA 

% Students % Students i Students 

701 or better 80% or b'^tter 80% or better 

Media before remed. before remed. after remed. 



CAI 

Audiotape -IP 
Syndactic text 
Linear text 
LAS 

Audio- video cape 



94.9 
89.4 
80.7 
74.1 
61.7 
69.9 



81.2 
74.2 
61.7 
53.7 
49.2 
44.3 



89.8 
87.5 
84.1 
68.0 
85.1 
63.9 



It can be noted from Table 7.2 that only segments - 
vdthin-CAI met the average 80% -80% criterion before 
remediation. However, audiotape-IP and syndact5c text 
segments met the average 80% -80% criterion fo.llov.'ing 
remediation. In reviewing these results, it must be 
rcmcmbci-ed that individual segments -wi thin-media and 
the number of test iteias per segment influence the 



pcrccnt-pcrcent averages* For example, three inefrective 
segments lowered the average percent of students 
achieving 80% in segments-vithin-linear text. (See 
Appendix V: Linear Text, Part Five.) Likewise, too, fexv' 
items on progress checT:s influenced the percentage of 
students responding to 80S of the items. (See Appendix V 
Syndactic Text, Segments 5.1, 5.2, .and 11.3.) Since 
fev;er than ten items were used in these segments, 805 
of the students could miss only one item on the test 
without automatically lowering the percent -percent 
criterion to less than 805-80%. 

Initial identification of segments requiring revision 
was based on the percent -percent distributions. The 
actual use of percent -percent data in making revisions 
is explained in Section VIII. 
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VIII. USE OF EFFliCTIVENRSS DATA IN COURSE IMPROVEMENT 

In addition to being simple indices of course 
effectiveness, the first implementation performance 
data were used as the basis for identification and 
revision of core materials and tests. As discussed 
in previous sections, the overall results of the first 
validation were quite good. Since 97% of the students 
met 70% of the total course objectives vdthout remediation, 
it is safe to conclude that the materials used verc at 
least minimally effective. On the other hand, since an 
optimally effective instructional system is one in 
which all students achieve, all objectives, revision of 
all ineffective segments and test items vas undertaken 
in order to approximate this criterion. 

MATERIALS REVISION 

Basic steps in the use of performance data for course 
revision w^ere: 

1. The identification of segmentr, x-^ithin media 
which fell below 801, i.e., averaged over media. 

2. The identification of each core segment v;hich 
fell below the 80% - 80% criterion level, i.e., 

80% of the students achieving 80% of the objectives. 

. % 



3. The identification of each objective v/ithin 

segments which was not passed by 805 or more 

of the students. 
.4. The identification of student responses to 

each test item alternative. 
5. The identification of student responses to 

each criterion frame within a programed 

sequence. 

Id entification of Media 

Results of segments -wi thin-media effectiveness are 
summarized in Section YII. Results indicated that no 
media was totally ineffective, but segments of LAS, 
videotape an . =iudiotape were somev;hat less effective 
and less interesting than CAI, audiotape-IP, and 
s>'TidaCi.ic text. 

Since differences in effectiveness between media 
indicated differ^^nces in materials preparation via media 
rather than differences inherent in the media themselves 
the question became "Is it possible to revise materials 
in these media in such a v/ay that all students will 
attain all objectives?". The determination v/as that 
little could be done to revise LAS segments since they 
were primarily taught by textboolcs, and revision of 
videotapes (as^they were used) v/as simply i\pt practical. 



As a result, the four segments of 3earn5ng aC'ivity 
sun^?naries were structured into segments of syndactic 
text, and videotapes were dropped from core segments. 
(Videotapes of actual leadership situations i^'ill be 
used in depth core segments in the next implementation, 
but videotapes of simple lectures i^ill not be used in 
core segments.) 

Although aud5.otapes were someiv'hat less effective 
than CAI, audiotape -IP, and syndactic text, they were 
revised and will be used in the next implementation. 
Reasons for dropping videotapes and not dropping audio- 
tapes v;ere that a transient presentation i^'as needed, 
for testing hypothesis 1, and audiotapes were less 
expensive to revise. Audiotapes ivill be parralleled 
by a persistent (text) form of presentation in the 
second implementation, and differences between audio-' 
tapes and text presentations will be experimentally 
studied. 

Identification of Segments 

Following identification of media which should be 
eliminated from the system, was identification of 
ineffective segments, irrespective of media. For 
example, in three linear text segments, the average 
percentages of students achieving 80?; of the objectives 



were 25.0?;, 4.5%, and 13.6-0. A review of these segments 
revealed that they should be re-forinatted 5nto smaller 
steps and that lengthy examples should be eliminated. 
The content outlines for these segments were also revised 
according to the USNA instructor's recommendations. Otho 
segments requiring substantial revision were identified 
and reviewed in a similar manner. 

Identification of Objective s 

In all, 751 of the segments required some revision 
according to the 80% - 801 criterion. The extent of the 
revision was based on the percentage of students 
responding to ^ach objective. For example, a segment 
falling below the 801 - 80% criterion may be generally 
ineffective or there may be only two or three objectives 
which are poorly presented or tested. An example of 
general ineffectiveness is Part Five, Segment IV, 
presented by linear text (Table 8.1). 



TABLB 8.1 



TEST ITKM'-OBJECTIVE REFERENCE 
(PART FIVE, SEGMENT IV) 



Objective 
identifier 



Test item 
number 



Percentage 
correct responses 
before remediation 



Percentage 
correct responses 
after remediation 



TO-l 


.1 


65.9 


72.7 


TO-2 


2 


84.1 


93.2 


TO- 3 


3 


.29.5 


45.5 


TO-4 


4 


79.5 


84.1 


TO- 5 


5 


100.0 


100.0 


TO- 6 


6 


90.9 


97.7 


TO- 7 


7 


25.0 


56.8 


TO- 8 


8 


47.7 


63.6 


TO-9 


• 

9 


79.5 


90.9 


TO-ll 


10 


93.2 


95.5 


T(j-ii 


li 


61.4 


68.2 



It can be noted that, before remediation, seven out 
of 11 text items were answered by less than 80? of the 
students. Follov/ing reraediation j five of these items 
were still below the 80% criterion. The distribution 
indicated that student performance was generally poor 
(63% of the objectives v:ere bclov; criterion), and the 
segment should be revised considerably. 

The example of general ineffectiveness can be 
contrasted with an example of specific ineffectiveness 
in which 80% of the students perform well on most 
objectives but do miss two or three of them. 

An example of specific ineffectiveness is Part Six, 
Segment IV, presented by syndactic text (Table 8.2). 



TABLE 8.2 



TEST ITEM-OBJECTIVE REFERENCE 
(PART SIX, SEGMENT IV) 

Percentage Percentage 
Objective Test item correct responses correct responses 
identifier num>or before remediation after remediation 



TO-l 


1 


95. S 


97.7 


TO-2 


2 


100.0 


100.0 


TO-3 


3 


93.2 


97.7 


TO-4 


4 


95.5 


97.7 


TO- 5 


5 


100.0 


100.0 


TO-7 


7 


97.7 


100.0 


TO-9 


9 


65.9 


79.5 


TO-10 


10 


52.3 


63.6 



Six out of eight items were answered correctly by 
901 or more of the students; only two were answered by 
less than 80% of the students. Therefore, the specific 
areas of difficulty in this syndactic text segment were 
the last two objectives, or last. two test items. Since 
students did- not improve appreciably on the test items 
after remediation, and since they remediated through 
the same core materials, the isolation of specif 
problems, i.e., test item or materials, is not apparent 
from the test item-objective reference alone. 

Identification of T est Item 

The general criterion for revision of either test 
items or materials was the 80% difficulty level of an 
item. ;Vhen items fell below the 80? level, the test 
item, objective, and content were reviewed with respect 
to the accuracy of their interrelationship. IVhere 
discrepancies occurred, subjective judgment determined 
if the test item or the materials should be changed in 
order to achieve a closer correspondence to the objective 
Wjhe.^e no discrepancy appeared, the following procedures 
were implemented: 

The distribution of responses to eacli distractor was 
studied in relation to the materials in order to detemin 
a) if the distractors were in fact correct, b) if the 



distractors were ipcorrect but not presented as distinctly 
incorrect, or c) if the correct answer was not emphasized 
as correct • 

Revision based on tliese considerations generally 
consisted of strengthening the controlling stimulus for 
the correct response by increasing the similarity of 
examples in the materials and the example used in the 
text item, or vice versa'. Wliere incorrect distractors 
were partially or actually correct, subject matter 
experts made decisions on revising the content outline 
or material. 

When available, the discrimination index of each 
item was used in conjunction with the difficulty ]ov^,i 
in deciding revision. Although the difficulty level 
was weighed more heavily in revision, all negatively 
discriminating items regardless of difficulty indicated 
need for revision. Since negative discrimination 
indicated that students who scored high or all other 
items were failing an item, the materials v;ere reviewed 
to see if in fact the correct answer was the only correct 
response or if tliere was another distractor v;hich was 
also correct. If the difficulty level was 90% or above, 
a discrimination index was not considered, since ojily 
two or three students could cause the discriminatioi^ 
index to be negative. However, if the difficulty ranged 



between 80 and 90 and the discriia.ination was negative, 
the nerd for revrlsioii to either test item or materials 
was clear. 

Although the discrimination indices were not 
available until after many test items and materials 
were revised, all progress checks were checked to 
determine if items that were negatively discriminating 
or significantly positively discriminating were revised 
under the difficulty rule. General guidelines for use 
of both difficulty and discrimination in revision are 
given in Table 8.3. 

TABLE 8.3 

REVISION DECISIONS BASED ON PROGRESS CHECK DATA 



Range of 
difficulty 



90-100 



80-90 



60-80 



0-60 



Discrimination 


Revision 


Test 
item 


Materia] 


+ 




No 


No 


No 






No 


No 


No 


0 




No . 


No 


No 


+ 




No 


No 


No 






Yes 


? 




0 




No 


No 


No 


+ 




Yes 


? 


Yes 






Yes 


Yes 


Yes 


0 




Yes 


•? 


? 






Yes 


? 


? 






Yes 


? 




0 




Yes 


7 





r 



Since data on item discrimination was not available 
vntil late in the revision cycle, it was not possible to 
fully implement these decision rules and verify their 
usefulness in the revision process. However, in subsequent 
revisions, these data will be used as additional 
guideline's to writers. 

I den t i f i c at i on o f F ?;ame s 

When materials required revision, writers analyzed 

material by criterion frames within the program sequences. 
Wlien test items required revision, writers revised items 
according to the item response analysis, item difficulty 
and item discrimination index. In materials revision, 
frame analysis indicated which criterion objectives 
included in the program sequence were ineffective. 
(An example of the frrme analysis format for syndactic 
text is given in Appendix K.) High and low error rates 
for both summary quizzes and frames of syndactic text 
were indicated on the frame analysis sheet. From these 
data, the writer could determine whether prerequisite 
OT ;rmina1 behaviors were failed by most students ♦ 

Theoretical decision rules for frame revision were 
supplemented by subjective judgment concerning the 
appropriateness of examples and clarity of presentation 
'being taught in weak frames. 



T13ST I'fKM UI-V3SI0N' 

Revision of progress checks y;as based on the 801 
difficulty level, the itejn :Tiscriminatjon index, and 
the item response analysis. Since these procedures, 
have already been indicated in Table 8.3, this section 
is limited to revision procedures for the administrative 
and cumulative posttcsts. 

Administra^^ive^Test Revisipn ^v** . 

wThe following guidelines were given to writers 
for the first revision of administrative test items: 
!)• Identify and revise all items which are above 
75% difficulty on the pretest. 

2) Identify and revise all items which have less 
than 75% difficulty if the difficulty of the 
corresponding posttcst item is less than 20% 
above that of the pretest item. 

3) Identify and revise items if the posttest difficulty 
is less than 70%. 

4) Under condition 2 above, revise all Items which 
have negative discrimination indices or which 
have positive discrimination indices that are^ 
significant, 

5) >vherever possible, revdse administrative items 
at the ?amo time the corresponding materials and 
progress check items are 'revised, and revise 
materials if they are tlie cause of negative 
discrimination or lev; difficulty* 



After all administrative items had been revised and 
were reviewed by WLC subject matter experts, the itemisj 
were pretested on two groups of USNA midshipmen, plebes 
and second classmen. Plebes v/ere tested in order to 
determine which items were simply too easy. Second 
classmen were tested in orde'r to determine iv'hich items 
measured objectives that second classmen had already 
learned. 

Specifications given to writers for the secojid 
revision of administrative test items i^ere: 

1) Work from second class to plebe data. Identify 
all items which 75% or more second classmen 
answered correctly. If 50% or more plebes also 
answered correctly, it is simply too easy and 
needs to be revised or replaced. If less than 
50% of the plebes answered correctly, leave it 
alone unless 80% or more of the second classmen 
answered correctly. In the latter case, replace 
the item. 

2) Work from plebe to second class data. Identify 
all items which 50% or more plebes answered 
correctly. If 66% or more second classmen also 
answered correctly, it is too easy and needs 
revision. 



3) . Use confidence data whore_it appears appropriate, 
i.e., if in doubt as to necessity for revision, 
check the confidence data for second classmen. 
The percentage of students getting the item 
correct but indicated no confidence in their 
response is the percentage who v/ere guessing. 
However, confidence data should in no way suggest 
necessity for revision, which the item response 
data does not suggest. * ^ 

* - . - 

Cumulative Posttest Revision 

Since CPT items were constructed to discriminate 

among students and among modules and since there was no 

t • 
pretest for CPTs, the guidelines for revision of CPTs 

differed from those for the administrative tests. 

Guidelines for CPT revision were: 

1) Identify all items which are over 85% difficulty 
or below 40% difficulty. These items will 
usually need to be revised considerably or 
replaced. 

2) Identify items which are betv/een 75% - 85% 
difficulty if they contain one distractor which 
has elicited no responses, and revise that 
distractor. 

3) Identify items v/hich are between 65% - 75% 
difficulty if they contain tv/o distractbxs 



which elicit no responses, and revise either 
of the distractors. 

4) Identify items between 40% - S0% if they 
contain a distractor which elicits more correct 
responses than the actyal correct response, 
and revise either the distractor or the 
correct alternatives. ^ . 

5) Identify and revise items between the 40% - 86% 
difficulty level that are negatively discriminating ' 
and t*ere not revised according to the difficulty 
rules. 

6) I£ an item has a significant positive discrimination 
index, do not revise it. 
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APPENDIX A 



COURSE OUTLINE 



PART ONE: Overview of Leadership 

Segmenf I ConcepJs of Leadership — * ' 

Segmeint II Standards of Leadership" In the Nayal Service 

PART TWO: Individual Behavior 



Segment I 
Segment II 
Segment III 
Segment IV 
Segmenl' V 
Segment Vl • 
Segment Vll 
Segment VIII 
Segment IX 



Introduction to Psychology 
Behavior and Its Observation 
Learning 

Factors Affecting Learning 
Attention and Perception 
Motivation 
Conflict 

Neurotic and Psychotic Reactions 
Personality ^ 



.1 



PART THREE: Group ^Dynamics 

Segment I Characteristics of Groups 

Segment il The Relation of the Leader to the Group 

Segment J II Group Interactions 

Segment IV , Conformity as a .Factor of Group Behavior 

Segment V Relation of the Individual to the Group 

PART FOUR: Achieving Effective Communication 

Segment I Importance of Interpersonal Communication 

Segment II Types of Communication 

Segment III The Communication Process (Receiver and Barriers) 

Segment iV The Communication Process (Sender and Feedback) 

Segment V Formal Communication and Its Dimensions 

Segment VI Informal Communication 

Segment Vll Communication Under Battle Situations 

PART fiVE: Military^ Manogement 



Segment I 
Segment II 
Segment 111 
Segment IV 
Segment V 
Segment VI 
Segment Vll 
Segment VIII 
Segment IX 
Segment X 



Introduction to Monagement and the Management Process 
Decision Making and Creativity 
Objectives 
Planning 
Organizing: 
Organizing: 
Organizing: 
Directing 
Controlling 
Coordinating 



Principles and Process 

Structure 

Charting 
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PART S!X: Aulhor ii y and Rosponsibiliiy 

-Secjment I Concept of Auihori^/ 

Segment II Wliy People Accept/Resist Authorhy 

Segment 111 . Delegcition of Authority; Line-^Staff Relationship 

Segment IV Responsibility 

PART SEVEN; Leadership Behavior and Style - 

Segment I Leadership, Behavior 

Segment II Leadership Style 

Segment III Determiners of Leadership SJyIe 

Segment IV Determiners of Leadership Style 

Segjnent V Participative Leadership • 

PAPJ EIGHT: Senidf-Sybordinate Relaiipriships ' 

Segment I Orgdhizational Structure & Social Distance in Senior- 

Subordinale Rejatiohships 

Segment Jl Officers-Enlisted Relation 

Segment III Assumption of Command and Formal & Informal Leader Relationships 

Segment IV Introduction to Counseling ^ 

Segm.ent V The Counseling Process 

Segment VI Relations with" Seniors and Contemporaries 



The Leader 

The Group and The Situalion 



PART NINE: Morale - Esprit de Corps 

Segment I Morale 

Segment II ' Group Solidarity and Esprit 

PART TEN; , DisctplSne' . 

Segment I j Introduction to Discipline 

Segment II Development and Maintenance of Discipline 

PART ELEVEN; Personnel Evaluation 

Segment I The Role of Evaluation 

Segment 11 Enlisted Performance Evaluation 

Segment III Officer Evaluation 



PART TWELVE; Applied' Leadership 

Segment I Measurement of Effective Leadership 

Segment 11 Generally Recognized Characteristics of an Effective Leader 

Segment III Techniques of Assuming Comrr.ond 

Segment IV "That's an Order!" 
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GUIDELINES FOR DEVELOPMENT OP BEHAVIORAL OBJECTIVES 

^' Type. 1 -- Definition - Identification 

Given the instruction ,to identify the correct 
purpose of (definition of/description of/use of) 
concept X, the M will select from several choices 
the correct purpose of (definition of/description 
of/use of) concept X. 

1. General Type 1 

a. Given the instruction to define 
concept X, 

' b. The M will select from several 

choices a definition of X 
■ c. Similar to the following: !'X....." 
(NOTE: The third part is optional.) 

2. Example of Typo 1 , 

-a. When given the instruction to 
define "acquisition," 

b. The M will select from several 
choices the correct definition. 

3» Example of Type 1 ^ ' . 

a. Given several choices,- the M will 
. • select the correct definition of 
acquisition. 



• 4. Example of Type 1 

a-. When asked to define "attention, 

b. The M will select from several 
choices a statement 

c. (which indicates that attention 
is the selection of specific 
stimulus elements) . 

Type 2 Discrimination - Comparison 
Given the instruction to evaluate the 
relationship between (defining attributes 
of /contrast between/comparison among) 
classes X, Y, Z...N, the M will be able 
to select from several choices the para- 
graph which, illustrates (describes/ 
differentiates/identifies) the relation- 
ship between (defining attributes of/ 
contrast between/comparison among) classes 
X, Y, Z . . .N. 
1. General- Typo I 

a. When asked to evaluate the 

relationship" -among X, Y,...N, 

b. The M mxW select from several 

choices the paragraph which 
describes this relationship. 



2. Example of Type 2 

a* When asked to evaluate the 

difference between retroactive 
and proactive inhibition, 
The M will select from several 
choices the paragraph which 
describes this difference* 

"Typo 3 Gpneralization - Problem Identification 

!• Deductive 

. a* Given examples of X, the M will 
be able to select from several 
choices the example which 
illustrates principle Y, 

2. Inductive 

a» Given an example of X, the M 
will be able to select from 
several choices which principle 

Y,*.*or N) is shown 
(exemplified/demonstrated) by 
the example* 

3. Genural Type 3 

a. Given examples of an X, 



b. The M will be able to select from 
several choices the example which 
illustrates principle Y. 

4* Example of Type 3 

a* When asked to compare several 
versions of the same communi- 
cation, 

h. The M will select from several 

choices the version which clearly 
links the subordinate's role to 
the overall objective of Naval 
operations. 
Type 4 Problem Solving 

Khen asked to evaluate a situation w^hich is an 
example of class X, the M will select from . 
several choices the correct solution (approach/ 
method/resolution of/roaction to) the situation 
using principle Y. 
1. General Type 4 

a. Given a problem situation which is 
an example of class X, 

b. The M will select from several 
choices the correct approach to the 
situation, 
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til 



c. Using principle Y. * 

(NOTE: The third patt may be omitted 

if .the objective is unambiguous. In 

practive, the third part is usually 

not given to the student.) 

Example of Type 4 

a. Wien asked to evaluate a situation 
in which there is an apparent 
failure in communication, 

b. ' The M \^ill select from several 

choices the des^iption which 
indicates the appropriate action 
of a leader 

c. Who assumes responsibility for. 
the failure. 
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Appendix C 
OUTLINE OF COURSE STRUCTURE AND MEDIA 



Part and " • 

Segment CPT 

Number Content Heading Unit^ Medium 

■PART ONE: OVERVIEW OF LEADERSHIP - 

I.I Concepts of Leadership* NR ST 

J. 2 Standards of Leadership In the Naval Service NR F-GD 

PART TWO: INDIVIDUAL BEHAVIOR 

2.1 Introduction to Psychology . • NR ST 

2.2 Behavior and Its Observation I AT-- or VT-PB 

2.3 Learning. j AT- or VT-PB 

2.4 Factors Affecting Learning I AT- or VT-PB 

2.5 Attention and Perception j AT- or VT-PB 

2.6 ^ Motivation 2 ST 

2.7 ' Conflict ^ 2 ST 

2.8 Neurotic and Psychotic Reactions 2 ST 
2.9' Personality NR LAS 

PART THREE: GROUP DYNAMICS 

3.1 Characteristics of Groups 3 AT- or VT-PB 

3.2 , The Relationship of the Leader to the Group 3 AT- or VT-PB 

3.3 . Group Interactions "5 AT- or VT-PB 

3.4 Conformity as a Factor of Group Behavior 3 AT- or VT-PB 

3.5 Relation of the Individual to the Group NR ■ ST 

PART FOUR: ACHIEVING EFFECTIVE COMMUNICATION 

4.1 Importance of Interpersonal Communication 

4.2 Types of Communication 

4.3 The Communication Process (Receiver and Barriers) 

4.4 The Communication Process (Sender and Feedback) 

4.5 Formal Communication and Its Dimensions 

4.6 Informal Communication 

4.7 Communication Under Battle Situations 

PART FIVE:' MILITARY MANAGEMENT 

5.1 Introduction to Management and the Management 

Process NR ST 

5.2 Decision Making and Creativity * NR ST 

5.3 Objectives NR ST 

5.4 Planning 6 LT 

5.5 Organizing: Principles and Process 6 LT 

5.6 Organizing: Structure ^ * • 6 LT 

5.7 Organizing: Charting 7 AT- or VT-PB 



4 


IT 




4 . 


IT 




4 


ir 




5 


/.'■ 


IPB 


5 


/. 


!P3 


5 






5 




<f'3 



•Part and 
Segment 

Number Content Heading 



5.8 
5.9 
5.10 



6.1 
6.2 
6.3 
6.4 



7.1 
7.2 
7.3 
7.4 



8.1 

8.2 
8.3 

8.4 

6.5 
8.6 



PART FIVE: MILITARY MANAGEMENT (CON'T) 
D.i rect 1 ng 
Control I ing 
Coordinating 



CPT 

Unit^ Medium 



7 AT- or VT-PB 
7 AT- or VT-PB 
7 AT- or VT-PB 



PART SIX: Authority and responsibility 

Concept of Authority 
Why People Accept/Resist Authority ^ 
Delegation of Authority; Line-Staff 
Resppnsibi I ity 



8 ST 

8 ST 

Relationship 8 ST 

NR ST 



PART SEVEN: LEADERSHIP BEHAV I OR ANCf STYLE 
Leadership Behavior 
Leadership Style 

Determiners of Leadership Style - The Leader 
Determiners of Leadership Style - The Group 

and The Situation 



7.5 Participative Leadership 



PART EIGHT: SENIOR-SUBORDINATE RELATIONSHIPS 
Organizational Structure i Social Distance in 

Senior-Subordinate Relationships 
Officer-Enlisted Relationships 
Assumption of Command and Formal i Informal 

Leader Relationships 
Introduction to Counseling 
The Counseling Process . 
Relations with Seniors and Contemporaries 



PART NINE: MORA.LE - ESPRIT DE CORPS 

9.1 Morale 

9.2 Group Solidarity and Esprit 

PART TEN: DISCIPLINE 

10. 1 Introduction to Discipline 

10.2 Development and Maintenance of Discipline 

PART ELEVEN: PERSONNEL EVALUATION 

11. 1 The Role of Evaluation 

11.2 Enlisted Performance Evaluation 
-11.3 -Officer Evaluation 



9 AT- or VT-PB 

9 AT- or VT-PB 

9 ■ AT- or VT-PB 

9 ■ AT- or VT-PB 

NR VT-PB 



10 LT 

■10 LT 

10 ■ LT 

11 LAS 
II LAS 
II LAS 



NR 
NR 



12 
12 
12 



NR VT-PB 
NR VT-PB 



.,AT-IP 
AT- IP 



ST 
ST 
ST 
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Part and 
Segment 

Number Content Heading 

PART TWELVE: APPLIED LEADERSHIP 
I2J Measurement of Ef f ective iLeadershIp 
12.2 General I y. Recognized Characteristics of an 

Effective Leader 
•12.3 Techniques of Assuming Command 
I2U "That^s an Order!" 



to: 



CPT 

Uhlt^ Medium 



13 


CA 


13 


CA 


13 


CA 


13 


CA 
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NR refers to a nonresearch segment, thus not assigned to a 
CPT unit. 

^ ST=SyndactIc (rnu Iti- level ) text; F-GD=Film, Group Discussion; 
AT=AudIotape; VT=Videotape; PB-=Panelbook; LAS=^Learning Activities 
Summary; LT=Llnear Text; I P= Intrinsic Program; CAI=Computer Assisted 
Instruction.. 
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APPENDIX E 

EXAMPLE: BEHAVIORAL OBJECTIVE AND CRITERION 
REFERENCED TEST ITEM' 
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APPENDIX }] 

EXAMPLE OF BEHAVIO.RAL OBJECTIVE: 

Given examples of leaders interacting with 
groups and using various leadership styles, 
the M will be able to identify the situation 
in which the leader uses participative 
leadership, 

EXAMPLE OF CRITERION REFERENCED TEST ITEM: 

LT Henry was the type of officer who left 
no doubt in the minds of his subordinates as 
to his leadership qualifications. He was 
alv/ays in the area, insuring that his orders 
were being carried out. When there was time, 
he consulted with his leading petty officers 
to obtain their ideas and' recommendations and 
shared decision making with them. LT Henry 
made sure that credit was given to those who . 
deserved it. LT Henry took the responsibility 
for success or failure. 

From the choices below, identify the type 
of leadership style being u&cd by LT Henry. 

a. Participative 

b. Supervisory 

c. Authoritarian 

d. Shared- leadership 
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FREQUENCY DISTRIBUTION OF ITEM DIFFICULTY AND DISCRIMINATION 
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EXAMPLE: ITEM DISCRIMINATION STATISTICS 
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TEST ITEM STATISTICS SUM?IARY: 
ADMINISTRATIVE POSTTEST 
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APPENDIX J 



■ FREQUENCY DISTRIBUTION OF ITEM DIFFICULTY 
AND POSITIVE DISCRIMINATION INDICES: 
PROGRESS CHECKS 
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FREQUENCY DISTRIBUTION OF ITEM DIFFICULTY 
AND POSITIVE DISCRIMINATION INDICES: PROGRESS CHECKS 
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FREQUENCY DISTRIBUTION OF 
NEGATIVE ITEM DISCRIMINATION INDICES 
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FREQUENCY DISTRIBUTION OF NEGATIVE ITEM DISCRIMINATION INDICES 



Interval Negative discrimination 

range frequency 
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FREQUENCY DISTRIBUTION OF 
ADMINISTRATIVE PRE AND POSTTEST SCORES 
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FREQUENCY DISTRIBUTION OF ADM I N I STR/J I VE PRE AND POSTTEST SCORES 
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GAIN RATIOS FOR ALL STUDENTS BASED ON 
ADMINISTRATIVE PRE AND POSTTESTS 
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TOTAL i OBJECTIVES ACHIEVED BY EACH STUDENT BEFORE AND AFTER REMEDIATION 



Student 


% Objectives 
before remediation 


% JDbJectlves 
after remediation 


1 


80.6 


sn. 1 


2 


78,5 


84.0 


3 


77.9 


83.4 


4 


80.4 


86.0 


5 


74.0 


81 .3 


6 


70.6 


76.2 


7 


76.4 


83.4 


8 


75.3 


81.5 


9 


82.5 


85.3 


iO 


75.1 


83.2 


It 


78.9 


87.5 


12 


78.3 


84.3 


\5 


84.0 ^ 


90.8 


14 


78.3 


85.1 


15 


. , 77.4 


82.1 


16 


■ 75.5 


86.8 


17 


79.8 


85.8 


18 


80.4 


85.8 


19 


66.8 


77.0 
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* 

student 


% Objectives 
before remediation 


i Objectives 
after remediation 


59 


79. 1 


fin Q 


AO 


82.8 


85.8 


41 


70.4 


81.5 


42 


72.6 


80.4 


43 


80.2 

• 


85.5 


44- 


74.5 

0 


82.8 


Mean % 


n.i 

* 


83.8 
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SI:GM1:NT 
MODULE " 



NAMK 
IDif ■ 
DATE' 



Exact Time Spent on Instructional 
Material in Module: ^Iinutes. 

Answer the starred items only i£ applicable. 

Complete all but Question 9 after v/orking through instructional 
material. 

Turn in questionnaire after ojnpleting the Progress Check. 



From youY OKn point of view: 


High 


Above 
Avg 


Avg 


Below 
Avg 


Low 


!• Was the material interesting? 












2. Was the material difficult? 












3. Were the questions difficult? 












*4. Was the video interesting? 












*5. Was the video lecturer 
interesting? 












■ — — — — ' 

*6. Was the audio presentation 
interesting? 












*7. Was the audio ^lecturer 
interesting? 












*8. Was the programed sequence 
interesting? 












» : 

9. Was the progress check 
difficult? 

* i» - - ' 













10. If possible, specify where you had difficulty. 



11. Comments and Suggestions. 
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APPENDIX V 



PERCENT STUDENTS SCORING 70% OR 8 0% ON PROGRESS 
CHECKS BEFORE AND AFTER REMEDIATION BY SEGMENT WITHIN MEDIA 
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PERCENTAGE OF STL0ENT5 SCORING 10$ or ON PROGRESS CHECKS 
BEFORE AND AFTER REMEDIATION BY SEGMENT WITHIN MEDIA 

AUDIOTAPE-VIDEOTAPE 



% Studonls 



% Students 



% Students 



segment 
number 


Number of 
test items 


Number of 
students 


70!? or better 
before remod. 


80^ or bettor 
before remed. 


80^ or 
after r 


2.2 


10 


44 


63.6 


31.8 


70.5 


2.3 


10 


44 


56.8 


18.2 


56.8 


2.4 


ID 


.43 


72.1 


34.9 


72.1 


2.5 


10 


44 


52.3 


25.0 


63.6 


3.1 


10 


44 


54.5 


22.7 


45-5 


3.2 


7 


44 


61 .4 


20.5 


38.6 


.3.3 


10 


44 


77.3 


63.6 


84.1 


3.4 


10 


43 


93.0 


72.1 


95.3- 


5.7 


8 


44 


79.5 


52.3 


77.3 


5.8 


9 


44 


95.5 


77.3 


79.5 


5.9 


9 


43 


67.4 


39.5 


76.7 


5.10 


7 


43 


58.1 


20.9 


51 .2 


7.1 


8 


44 


59.1 


16.2 


43.2 


7.2 


4 


44 


88.6 


45.5 


75.0 


7.3 


9 


44 


59.1 


34.1 


45.5 


7.A . 


6 


44 


59.1 


59.1 


79.5 


7.5 


7 


44 


79.5 


43.2 


70.5 


9. 1 


5 


44 


68.2 


68.2 


79.5 


9.2 


5 


44 


93.2 


93.2 


100.0 



Unweighted mean % students 



69.9 



44.3 



63.9 
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ft 



AUDIOTAPE- IP BOOKLET 



Part and 

segment 

number 


Number of 
test Items 


Number of 
students 


i Students 
1Q% or better 
before remed. 


% Students 
80^ or better 
before remed. 


% Students 
80^ or better 
after remed. 


4.4 


10 


44 


97.7 


• 

95.5 


97.7 • 


4.5 


9 


44 


81 .8 


43.2 


54.5 


4.6 


io 


44 


72.7 

• 


52.3 


79.5 


4.7 


10 


44 


84.1 


68.2 


93.2 


iO. i 


7 . 


44 


100.0 


90.9 


100.0 


10.2 


10 


44 


100.0 


95.5 


100.0 


Unwelgh-fed mean % students 


89.4 


74.2 


87.5 
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CAI 

Part and % S+uden+s % Students % Students 

■ segment Number of Number of 10% or better 80^ or better 80^ or bettor 

number test items s tudents b efore remed, before remed, a f ter_ romed. 

12.1 10 44 95.5 84.1 93.2 

12.2 9 44 93.2 59.1 70.5 

12.3 10 43 100.0 97.7 100.0 

12.4 10 44 90.9 84.1 95.5 



Unweighted mean % students 



94.9 



81.2 



89.8 



LAS 





Part and 

segment 

number 


Number of 
test Items 


Number of 
students 


% Students 
70% or better 
before remed. 


% Students 
80% or betler 
before remed. 


% Students 
80^ or better 
after remed. 


2.9 


10 


- 44 


47.7 


■ 36.4 


79.5 


8.4 


8 


44 


65.9 


43.2 


79.5 


8.5 


11 


43 


76.7 


60.5 


93.0 


8.6 


6 


44 


56.8 


56.8 


88.6 



Unweighted mean % students 



61 .7 



49.2 



85.1 
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LINEAR TEXT 



Part and 

segfrient 

number 


NumDer or 
test Items 


iNumuur \jj 
students 


% Students 
10% or bettor 
before remed. 


% Students 
80? or better, 
before remed. 


i Students 
oO% or better 
after remed. 


4.1 


9 


44 


86.4 


61.4 


72.7 


4.2 


10 


44 


97.7 


88.6 


95.5 


4.3 


10 


44 


90.9 


88.6 


95.5 


5.4 


II 


44 


54.5 


25.0 


47.7 


5.5 


9 ■. 


44 


9.1 


4.5 


11.4 


5.6 


8 


44 


65.9 


13.6 


40.9 


3.1 


10 


44 


84.1 


70.5 


84.1 


8.2 


13 


42 


78.6 


42.9 


66.7 


8.3 


8 


44 


100.0 


88.6 


97.7 


Unweighted mean % students 


74.1 


53.7 


68.0 
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SYNOACTIC TEXT 

Part and % Students $ Students % Students 



segmoni' 
numoer 


Number of 

1 i 1 1 


Number of 

51 U06n 1 5 


70% or better 
before remed. 


80^ or better 
before remed. 


80^ or botlcr 
after remod. 


1. 1 


10 


44 


90.9 


70 5 


Q3 9 


2.1 


10 


44 


79.5 


65 9 


97 7 


2.6 


10 


44 


68.2 


40 9 


79 7 


2.7 


10 


44 


36.4 


1 1 4 


43 9 


2.8 


10 


41 


97 7 


66 4 


Q3 9 


3.5 


io 


43 


95 3 


83 7 


inn n 




9. 


44 


68.2 


43.2 


81.8 


5.2 


8 


. 44 


54.5 


22.7 


70.5 


5.3 


10 


44 


75.0 


52.3 


86.4 


6.1 


14 


44 


86.4 


50.0 


75.0 


6.2 


6 ' 


44 


72.7 


72.7 . 


90.9 


6.3 


12 


44 


{00.0 


95.5 


100.0 


6.4 


8" 


44 


95.5 


79.5 


95.5 


1 1 J 


10 


44 


• 84.1 


70.5 


84.1 


11.2 


8 


• 44 


100.0 


88.6 • 


90.9 


11.3 


4 


44 


86.4 


57,3 


70.5 



Unweighted mean % students 



80.7 



61.7 



84.1 



APPENDIX W 
EXAMPLE: SYNDACTIC TEXT FRAME ANALYSIS 



